[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:03 dist-test-slave-zfp8 krb5kdc[15778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:03 dist-test-slave-zfp8 krb5kdc[15778](info): set up 2 sockets
Jun 20 18:44:03 dist-test-slave-zfp8 krb5kdc[15778](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 18:44:04.139739 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.010s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:04 dist-test-slave-zfp8 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:04.311456 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37609
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:37609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:04.651057 15791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:04.651409 15791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:04.651507 15791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:04.658928 15791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:04.659073 15791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:04.659135 15791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:04.659204 15791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:04.659344 15791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:04.670403 15791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:37609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:04.671851 15791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:04.673110 15791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:04.685721 15796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:04.685832 15799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:04.685806 15797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:04.686322 15791 server_base.cc:1034] running on GCE node
I20260620 18:44:04.687389 15791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:04.688890 15791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:04.690125 15791 hybrid_clock.cc:648] HybridClock initialized: now 1781981044690077 us; error 49 us; skew 500 ppm
I20260620 18:44:04.690475 15791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:04 dist-test-slave-zfp8 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981044, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:04.693533 15791 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:04.695051 15791 webserver.cc:460] Webserver started at http://127.15.102.254:37351/ using document root <none> and password file <none>
I20260620 18:44:04.695809 15791 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:04.695955 15791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:04.696262 15791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:04.699331 15791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "6c825ef1b4634df6a934570c6b8eda3f"
format_stamp: "Formatted at 2026-06-20 18:44:04 on dist-test-slave-zfp8"
server_key: "d9c1760d392452735e64a52e616b0d13"
server_key_iv: "974bfa8bdbd6bdf3ff7b489924449f2b"
server_key_version: "encryptionkey@0"
I20260620 18:44:04.700140 15791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "6c825ef1b4634df6a934570c6b8eda3f"
format_stamp: "Formatted at 2026-06-20 18:44:04 on dist-test-slave-zfp8"
server_key: "d9c1760d392452735e64a52e616b0d13"
server_key_iv: "974bfa8bdbd6bdf3ff7b489924449f2b"
server_key_version: "encryptionkey@0"
I20260620 18:44:04.705899 15791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:44:04.709503 15806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:04.710901 15791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:04.711110 15791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "6c825ef1b4634df6a934570c6b8eda3f"
format_stamp: "Formatted at 2026-06-20 18:44:04 on dist-test-slave-zfp8"
server_key: "d9c1760d392452735e64a52e616b0d13"
server_key_iv: "974bfa8bdbd6bdf3ff7b489924449f2b"
server_key_version: "encryptionkey@0"
I20260620 18:44:04.711369 15791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:04.730712 15791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:04.735316 15791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:04.735615 15791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:04.755693 15791 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:37609
I20260620 18:44:04.755715 15857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:37609 every 8 connection(s)
I20260620 18:44:04.757107 15791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:04.759280 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 15791
I20260620 18:44:04.759611 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:04.762518 15858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:04.762686 15771 external_mini_cluster.cc:1428] Setting key f3eb5c27130e7859744e8f044b412739
I20260620 18:44:04.775532 15858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f: Bootstrap starting.
I20260620 18:44:04.778795 15858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:04.780026 15858 log.cc:824] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:04.783087 15858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f: No bootstrap required, opened a new log
I20260620 18:44:04.790612 15858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c825ef1b4634df6a934570c6b8eda3f" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37609 } }
I20260620 18:44:04.791023 15858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:04.791142 15858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c825ef1b4634df6a934570c6b8eda3f, State: Initialized, Role: FOLLOWER
Jun 20 18:44:04 dist-test-slave-zfp8 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:04.791849 15858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c825ef1b4634df6a934570c6b8eda3f" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37609 } }
I20260620 18:44:04.792105 15858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:04.792253 15858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:04.792438 15858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:04.795521 15858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c825ef1b4634df6a934570c6b8eda3f" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37609 } }
I20260620 18:44:04.796120 15858 leader_election.cc:302] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c825ef1b4634df6a934570c6b8eda3f; no voters:
I20260620 18:44:04.796818 15858 leader_election.cc:288] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:04.796955 15863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:04.799078 15858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:04.803978 15863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [term 1 LEADER]: Becoming Leader. State: Replica: 6c825ef1b4634df6a934570c6b8eda3f, State: Running, Role: LEADER
I20260620 18:44:04.804566 15863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c825ef1b4634df6a934570c6b8eda3f" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37609 } }
I20260620 18:44:04.807498 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:04.767440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.15.102.254:37609)
0620 18:44:04.768577 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:04.768624 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:04.769747 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 18:44:04.778385 (+ 8638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:04.778409 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:04.778825 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:04.779348 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:04.782833 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:04.784282 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:04.786254 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:04.786959 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:04.787228 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:04.793181 (+ 5953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:04.793190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:04.793208 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:04.793254 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:04.797381 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:04.798862 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:04.798875 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:04.798888 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:04.799191 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:04.799838 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:04.799846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:04.799852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:04.800810 (+ 958us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:04.803614 (+ 2804us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:04.804021 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 18:44:04.804754 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":416,"threads_started":1}
I20260620 18:44:04.808168 15865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c825ef1b4634df6a934570c6b8eda3f. Latest consensus state: current_term: 1 leader_uuid: "6c825ef1b4634df6a934570c6b8eda3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c825ef1b4634df6a934570c6b8eda3f" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37609 } } }
I20260620 18:44:04.808776 15865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:04.809759 15864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c825ef1b4634df6a934570c6b8eda3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c825ef1b4634df6a934570c6b8eda3f" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37609 } } }
I20260620 18:44:04.810129 15864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:04.815595 15873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:04.821720 15873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:04.831553 15873 catalog_manager.cc:1347] Generated new cluster ID: 86c751db1af74e5bb83c963d6dc68edb
I20260620 18:44:04.831683 15873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:04.860443 15873 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:04.862075 15873 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:04.879716 15873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c825ef1b4634df6a934570c6b8eda3f: Generated new TSK 0
I20260620 18:44:04.880657 15873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:04.928802 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37609
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:05.272794 15884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:05.273094 15884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:05.273272 15884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:05.280699 15884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:05.280820 15884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:05.280937 15884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:05.291139 15884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:05.292471 15884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:05.293846 15884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:05.308220 15889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:05.308274 15892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:05.308908 15890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:05.309110 15884 server_base.cc:1034] running on GCE node
I20260620 18:44:05.309783 15884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:05.310916 15884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:05.312141 15884 hybrid_clock.cc:648] HybridClock initialized: now 1781981045312069 us; error 65 us; skew 500 ppm
I20260620 18:44:05.312470 15884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:05 dist-test-slave-zfp8 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:05.322921 15884 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:05.324666 15884 webserver.cc:460] Webserver started at http://127.15.102.193:37357/ using document root <none> and password file <none>
I20260620 18:44:05.325415 15884 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:05.325567 15884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:05.325889 15884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:05.329038 15884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "0803146c03e04fedb857a354fa0ecb52"
format_stamp: "Formatted at 2026-06-20 18:44:05 on dist-test-slave-zfp8"
server_key: "9764f1520459265f9f269307826f4b4a"
server_key_iv: "b070bfff0446d68dba92361f3f3c71a9"
server_key_version: "encryptionkey@0"
I20260620 18:44:05.329813 15884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "0803146c03e04fedb857a354fa0ecb52"
format_stamp: "Formatted at 2026-06-20 18:44:05 on dist-test-slave-zfp8"
server_key: "9764f1520459265f9f269307826f4b4a"
server_key_iv: "b070bfff0446d68dba92361f3f3c71a9"
server_key_version: "encryptionkey@0"
I20260620 18:44:05.335002 15884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:44:05.338459 15899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:05.339870 15884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:05.340045 15884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "0803146c03e04fedb857a354fa0ecb52"
format_stamp: "Formatted at 2026-06-20 18:44:05 on dist-test-slave-zfp8"
server_key: "9764f1520459265f9f269307826f4b4a"
server_key_iv: "b070bfff0446d68dba92361f3f3c71a9"
server_key_version: "encryptionkey@0"
I20260620 18:44:05.340253 15884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:05.374238 15884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:05.378966 15884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:05.379336 15884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:05.380296 15884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:05.381628 15884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:05.381742 15884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:05.381901 15884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:05.381973 15884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:05.418821 15884 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:44465
I20260620 18:44:05.418869 16011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:44465 every 8 connection(s)
I20260620 18:44:05.420583 15884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:05.425992 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 15884
I20260620 18:44:05.426234 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:05.426826 15771 external_mini_cluster.cc:1428] Setting key bd4edb782e730c75b50cb92da8456160
Jun 20 18:44:05 dist-test-slave-zfp8 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:05.440968 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:05.423156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:46657 (local address 127.15.102.254:37609)
0620 18:44:05.423610 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:05.423615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:05.424624 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 18:44:05.426526 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:05.426531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:05.426621 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:05.426747 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:05.428346 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:05.429084 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:05.430627 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:05.431018 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:05.431155 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:05.435447 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:05.435452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:05.435455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:05.435486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:05.437927 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:05.438677 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:05.438682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:05.438686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:05.438802 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:05.439266 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:05.439273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:05.439279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:05.439534 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:05.439690 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:05.440382 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 18:44:05.440609 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
I20260620 18:44:05.442018 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:05.423808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37609 (local address 127.15.102.193:46657)
0620 18:44:05.424450 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:05.424493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:05.426216 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:05.426997 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:05.427009 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:05.427605 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:05.428169 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:05.428188 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:05.429252 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:05.429257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:05.430474 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:05.430485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:05.431287 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:05.431297 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:05.431529 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:05.432300 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:05.432342 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:05.435185 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:05.438086 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:05.438095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:05.438113 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:05.438544 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:05.438916 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:05.438921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:05.438925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:05.439114 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:05.439665 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:05.439672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:05.440019 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 18:44:05.440287 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 18:44:05.440821 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":262,"threads_started":1}
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:44:05.443607 16012 heartbeater.cc:347] Connected to a master server at 127.15.102.254:37609
I20260620 18:44:05.443943 16012 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:05.444777 16012 heartbeater.cc:511] Master 127.15.102.254:37609 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
I20260620 18:44:05.447260 15823 ts_manager.cc:194] Registered new tserver with Master: 0803146c03e04fedb857a354fa0ecb52 (127.15.102.193:44465)
I20260620 18:44:05.449316 15823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:46657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:05.466926 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37609
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:05.802078 16018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:05.802361 16018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:05.802459 16018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:05.809978 16018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:05.810102 16018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:05.810217 16018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:05.820384 16018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:05.821728 16018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:05.823143 16018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:05.836079 16023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:05.836109 16026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:05.836218 16024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:05.837755 16018 server_base.cc:1034] running on GCE node
I20260620 18:44:05.838408 16018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:05.839406 16018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:05.840600 16018 hybrid_clock.cc:648] HybridClock initialized: now 1781981045840545 us; error 62 us; skew 500 ppm
I20260620 18:44:05.840914 16018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:05 dist-test-slave-zfp8 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:05.844043 16018 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:05.845448 16018 webserver.cc:460] Webserver started at http://127.15.102.194:44887/ using document root <none> and password file <none>
I20260620 18:44:05.846127 16018 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:05.846225 16018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:05.846464 16018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:05.849696 16018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "d35b84db01034586854aca2e5a4aaf40"
format_stamp: "Formatted at 2026-06-20 18:44:05 on dist-test-slave-zfp8"
server_key: "77f8888d18f53b8970072186244d2aaf"
server_key_iv: "ce5e1361ce75898fe25495e21aa1d2c7"
server_key_version: "encryptionkey@0"
I20260620 18:44:05.850379 16018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "d35b84db01034586854aca2e5a4aaf40"
format_stamp: "Formatted at 2026-06-20 18:44:05 on dist-test-slave-zfp8"
server_key: "77f8888d18f53b8970072186244d2aaf"
server_key_iv: "ce5e1361ce75898fe25495e21aa1d2c7"
server_key_version: "encryptionkey@0"
I20260620 18:44:05.855366 16018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:44:05.858721 16033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:05.860116 16018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:05.860280 16018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "d35b84db01034586854aca2e5a4aaf40"
format_stamp: "Formatted at 2026-06-20 18:44:05 on dist-test-slave-zfp8"
server_key: "77f8888d18f53b8970072186244d2aaf"
server_key_iv: "ce5e1361ce75898fe25495e21aa1d2c7"
server_key_version: "encryptionkey@0"
I20260620 18:44:05.860471 16018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:05.875730 16018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:05.880002 16018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:05.880287 16018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:05.881227 16018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:05.883138 16018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:05.883209 16018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:05.883378 16018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:05.883442 16018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:05.918402 16018 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:36609
I20260620 18:44:05.918438 16145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:36609 every 8 connection(s)
I20260620 18:44:05.919852 16018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:05.923864 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 16018
I20260620 18:44:05.924055 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:05.924589 15771 external_mini_cluster.cc:1428] Setting key 5dd2a2a732df11a35a2d0bac0e670085
Jun 20 18:44:05 dist-test-slave-zfp8 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981045, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
I20260620 18:44:05.941521 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:05.922365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:53081 (local address 127.15.102.254:37609)
0620 18:44:05.922562 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:05.922568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:05.923913 (+ 1345us) server_negotiation.cc:407] Connection header received
0620 18:44:05.926149 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:05.926154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:05.926228 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:05.926335 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:05.928877 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:05.929553 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:05.931880 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:05.932314 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:05.932467 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:05.935872 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:05.935877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:05.935880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:05.935911 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:05.938434 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:05.939165 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:05.939172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:05.939176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:05.939355 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:05.939788 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:05.939793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:05.939796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:05.940073 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:05.940237 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:05.940958 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 18:44:05.941192 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:44:05.942512 16148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:05.922896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37609 (local address 127.15.102.194:53081)
0620 18:44:05.923717 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:05.923779 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:05.925898 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:05.926574 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:05.926614 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:05.927858 (+ 1244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:05.928698 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:05.928728 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:05.929714 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:05.929722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:05.931736 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:05.931747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:05.932551 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:05.932558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:05.932792 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:05.933484 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:05.933506 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:05.935631 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:05.938591 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:05.938609 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:05.938627 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:05.939028 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:05.939466 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:05.939471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:05.939475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:05.939668 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:05.940210 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:05.940218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:05.940587 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 18:44:05.940863 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 18:44:05.941392 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":312,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
I20260620 18:44:05.944067 16146 heartbeater.cc:347] Connected to a master server at 127.15.102.254:37609
I20260620 18:44:05.944417 16146 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:05.945371 16146 heartbeater.cc:511] Master 127.15.102.254:37609 requested a full tablet report, sending...
I20260620 18:44:05.946973 15823 ts_manager.cc:194] Registered new tserver with Master: d35b84db01034586854aca2e5a4aaf40 (127.15.102.194:36609)
I20260620 18:44:05.947829 15823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:53081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:05.966050 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37609
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:06.306186 16151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:06.306468 16151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:06.306651 16151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:06.314703 16151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:06.314828 16151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:06.315049 16151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:06.325390 16151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:06.326773 16151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:06.328300 16151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:06.342728 16157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:06.342700 16156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:06.342973 16159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:06.343937 16151 server_base.cc:1034] running on GCE node
I20260620 18:44:06.344606 16151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:06.345739 16151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:06.346972 16151 hybrid_clock.cc:648] HybridClock initialized: now 1781981046346926 us; error 42 us; skew 500 ppm
I20260620 18:44:06.347352 16151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:06 dist-test-slave-zfp8 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:06.350358 16151 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:06.352281 16151 webserver.cc:460] Webserver started at http://127.15.102.195:38637/ using document root <none> and password file <none>
I20260620 18:44:06.352973 16151 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:06.353116 16151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:06.353436 16151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:06.356407 16151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "389e5e5f02684bb186f1f77b86ba4d2e"
format_stamp: "Formatted at 2026-06-20 18:44:06 on dist-test-slave-zfp8"
server_key: "31351e97816f2ab293009fe0da36a7cc"
server_key_iv: "1a1ab453add51403ae09ccb25e358fc7"
server_key_version: "encryptionkey@0"
I20260620 18:44:06.357167 16151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "389e5e5f02684bb186f1f77b86ba4d2e"
format_stamp: "Formatted at 2026-06-20 18:44:06 on dist-test-slave-zfp8"
server_key: "31351e97816f2ab293009fe0da36a7cc"
server_key_iv: "1a1ab453add51403ae09ccb25e358fc7"
server_key_version: "encryptionkey@0"
I20260620 18:44:06.362306 16151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:44:06.366008 16166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:06.367419 16151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:06.367584 16151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "389e5e5f02684bb186f1f77b86ba4d2e"
format_stamp: "Formatted at 2026-06-20 18:44:06 on dist-test-slave-zfp8"
server_key: "31351e97816f2ab293009fe0da36a7cc"
server_key_iv: "1a1ab453add51403ae09ccb25e358fc7"
server_key_version: "encryptionkey@0"
I20260620 18:44:06.367777 16151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:06.387413 16151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:06.391897 16151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:06.392248 16151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:06.393270 16151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:06.394608 16151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:06.394722 16151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:06.394871 16151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:06.394946 16151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:06.431701 16151 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:35193
I20260620 18:44:06.431723 16278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:35193 every 8 connection(s)
I20260620 18:44:06.433032 16151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:06.443490 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 16151
I20260620 18:44:06.443706 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:06.444236 15771 external_mini_cluster.cc:1428] Setting key 1b1f34bdab450098b92ab5caf01c8de6
Jun 20 18:44:06 dist-test-slave-zfp8 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981046, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:06.452430 16012 heartbeater.cc:503] Master 127.15.102.254:37609 was elected leader, sending a full tablet report...
I20260620 18:44:06.455909 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:06.435561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:36931 (local address 127.15.102.254:37609)
0620 18:44:06.435833 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:06.435838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:06.437107 (+ 1269us) server_negotiation.cc:407] Connection header received
0620 18:44:06.439356 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:06.439362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:06.439455 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:06.439570 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:06.441053 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.441807 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.443410 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.443942 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.444073 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:06.448325 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:06.448331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:06.448336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:06.448376 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:06.451218 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.452419 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.452424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.452428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.452631 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.453166 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.453173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.453178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.453583 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:06.453764 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:06.455379 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0620 18:44:06.455600 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 18:44:06.456674 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:06.436046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37609 (local address 127.15.102.195:36931)
0620 18:44:06.436849 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:06.436893 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:06.439025 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:06.439805 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:06.439813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:06.440332 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:06.440875 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:06.440892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.441958 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.441964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:06.443211 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:06.443280 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.444195 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.444202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:06.444500 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:06.445510 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:06.445543 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:06.448067 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:06.451412 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:06.451422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:06.451443 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:06.452221 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:06.452760 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:06.452767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:06.452770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:06.453021 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:06.453759 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:06.453766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:06.454285 (+ 519us) client_negotiation.cc:769] Sending connection context
0620 18:44:06.454636 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 18:44:06.455197 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":282,"threads_started":1}
I20260620 18:44:06.458669 16279 heartbeater.cc:347] Connected to a master server at 127.15.102.254:37609
I20260620 18:44:06.458940 16279 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:06.459829 16279 heartbeater.cc:511] Master 127.15.102.254:37609 requested a full tablet report, sending...
I20260620 18:44:06.461359 15823 ts_manager.cc:194] Registered new tserver with Master: 389e5e5f02684bb186f1f77b86ba4d2e (127.15.102.195:35193)
I20260620 18:44:06.462182 15823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:36931
I20260620 18:44:06.473462 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:44:06 dist-test-slave-zfp8 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:44:06 dist-test-slave-zfp8 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981046, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:06.517951 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:06.501573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44592 (local address 127.15.102.193:44465)
0620 18:44:06.502323 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:06.502334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:06.502365 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:44:06.502502 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:06.502509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:06.502741 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:06.502910 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:06.504414 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.505459 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.506854 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.507498 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.507642 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:06.511541 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:06.511556 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:06.511594 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:06.511635 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:06.515076 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.515896 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.515904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.515906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.516128 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.516554 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.516557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.516559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.516920 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:06.517154 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:06.517446 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 18:44:06.517654 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":365,"threads_started":1}
W20260620 18:44:06.519311 15927 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44592
Jun 20 18:44:06 dist-test-slave-zfp8 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:44:06 dist-test-slave-zfp8 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:06.562175 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:06.548012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.15.102.193:44465)
0620 18:44:06.548301 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:06.548310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:06.548602 (+ 292us) server_negotiation.cc:407] Connection header received
0620 18:44:06.548823 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:06.548828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:06.548918 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:06.549030 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:06.550320 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.550994 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.552292 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.552702 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.552831 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:06.556698 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:06.556703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:06.556706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:06.556738 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:06.559737 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.560364 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.560370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.560378 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.560491 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.560847 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.560854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.560857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.561096 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:06.561330 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:06.561629 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:44:06.561846 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:44:06.563658 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 15884
I20260620 18:44:06.576442 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:06.564807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.15.102.193:44465)
0620 18:44:06.565006 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:06.565010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:06.565206 (+ 196us) server_negotiation.cc:407] Connection header received
0620 18:44:06.565319 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:06.565323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:06.565379 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:06.565471 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:06.566675 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.567462 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.568724 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.569141 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.569243 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:06.571601 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:06.571606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:06.571610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:06.571639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:06.574130 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.574715 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.574718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.574720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.574835 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.575140 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.575143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.575146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.575488 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:06.575674 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:06.575920 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 18:44:06.576068 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:44:06.577162 16007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:06.663481 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 15884
I20260620 18:44:06.679927 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16018
Jun 20 18:44:06 dist-test-slave-zfp8 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:44:06.695662 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:06.680810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.15.102.194:36609)
0620 18:44:06.681307 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:06.681314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:06.681339 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:44:06.681397 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:06.681403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:06.681612 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:06.681791 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:06.682966 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.684036 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.685272 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.685826 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.685970 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:06.690007 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:06.690019 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:06.690033 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:06.690065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:06.692832 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.693512 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.693519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.693522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.693645 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.694056 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.694062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.694065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.694397 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:06.694715 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:06.695013 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 18:44:06.695306 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":240,"threads_started":1}
I20260620 18:44:06.696895 16141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:06.779093 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 16018
I20260620 18:44:06.794571 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16151
Jun 20 18:44:06 dist-test-slave-zfp8 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:44:06.810392 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:06.795554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.15.102.195:35193)
0620 18:44:06.796107 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:06.796114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:06.796144 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:44:06.796204 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:06.796211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:06.796424 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:06.796554 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:06.797715 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.798734 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.800092 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:06.800745 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:06.800842 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:06.804822 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:06.804833 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:06.804849 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:06.804880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:06.807893 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.808571 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.808577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.808580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.808698 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:06.809086 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:06.809089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:06.809091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:06.809415 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:06.809630 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:06.809921 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:44:06.810110 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":270,"threads_started":1}
I20260620 18:44:06.811925 16274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:06.892822 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 16151
I20260620 18:44:06.908780 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 15791
I20260620 18:44:06.909976 15853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:07.010792 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 15791
2026-06-20T18:44:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3970 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:07 dist-test-slave-zfp8 krb5kdc[16300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:07 dist-test-slave-zfp8 krb5kdc[16300](info): set up 2 sockets
Jun 20 18:44:07 dist-test-slave-zfp8 krb5kdc[16300](info): commencing operation
krb5kdc: starting...
W20260620 18:44:08.095418 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:08 dist-test-slave-zfp8 krb5kdc[16300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:08.270208 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35403
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:35403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:08.587700 16313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:08.588071 16313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:08.588238 16313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:08.595422 16313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:08.595579 16313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:08.595692 16313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:08.595794 16313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:08.595878 16313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:08.605598 16313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:35403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:08.607012 16313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:08.608354 16313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:08.620376 16318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:08.620414 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:08.620612 16321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:08.621520 16313 server_base.cc:1034] running on GCE node
I20260620 18:44:08.622385 16313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:08.623970 16313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:08.625211 16313 hybrid_clock.cc:648] HybridClock initialized: now 1781981048625172 us; error 50 us; skew 500 ppm
I20260620 18:44:08.625669 16313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:08 dist-test-slave-zfp8 krb5kdc[16300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981048, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:08.628978 16313 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:08.630548 16313 webserver.cc:460] Webserver started at http://127.15.102.254:36999/ using document root <none> and password file <none>
I20260620 18:44:08.631371 16313 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:08.631532 16313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:08.631891 16313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:08.635025 16313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "8e3ede126e1342bba387386cee8c13c8"
format_stamp: "Formatted at 2026-06-20 18:44:08 on dist-test-slave-zfp8"
server_key: "a03b595fed39a0d07303ed4fcd5a6507"
server_key_iv: "4a78e01fff56529c6e253512b26474e7"
server_key_version: "encryptionkey@0"
I20260620 18:44:08.635923 16313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "8e3ede126e1342bba387386cee8c13c8"
format_stamp: "Formatted at 2026-06-20 18:44:08 on dist-test-slave-zfp8"
server_key: "a03b595fed39a0d07303ed4fcd5a6507"
server_key_iv: "4a78e01fff56529c6e253512b26474e7"
server_key_version: "encryptionkey@0"
I20260620 18:44:08.641248 16313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 18:44:08.644748 16328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:08.646168 16313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:44:08.646378 16313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "8e3ede126e1342bba387386cee8c13c8"
format_stamp: "Formatted at 2026-06-20 18:44:08 on dist-test-slave-zfp8"
server_key: "a03b595fed39a0d07303ed4fcd5a6507"
server_key_iv: "4a78e01fff56529c6e253512b26474e7"
server_key_version: "encryptionkey@0"
I20260620 18:44:08.646586 16313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:08.665750 16313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:08.670148 16313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:08.670449 16313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:08.688670 16313 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:35403
I20260620 18:44:08.688701 16379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:35403 every 8 connection(s)
I20260620 18:44:08.690073 16313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:08.693781 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:08.696556 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 16313
I20260620 18:44:08.696785 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:08.697590 15771 external_mini_cluster.cc:1428] Setting key 8a117375c7138afa5929c765e7704f2d
I20260620 18:44:08.706396 16380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8: Bootstrap starting.
I20260620 18:44:08.710192 16380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:08.711552 16380 log.cc:824] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:44:08 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:08.714548 16380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8: No bootstrap required, opened a new log
I20260620 18:44:08.721172 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:08.699757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.15.102.254:35403)
0620 18:44:08.700612 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:08.700625 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:08.700669 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:44:08.702295 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:08.702316 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:08.702719 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:08.703053 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:08.704846 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:08.706113 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:08.707832 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:08.708512 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:08.708645 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:08.713547 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:08.713556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:08.713569 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:08.713605 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:08.716733 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:08.717626 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:08.717635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:08.717642 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:08.717831 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:08.718279 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:08.718283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:08.718285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:08.718842 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:08.719077 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:08.719668 (+ 591us) server_negotiation.cc:299] Negotiation successful
0620 18:44:08.720087 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":390,"threads_started":1}
I20260620 18:44:08.723826 16380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3ede126e1342bba387386cee8c13c8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35403 } }
I20260620 18:44:08.724264 16380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:08.724376 16380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3ede126e1342bba387386cee8c13c8, State: Initialized, Role: FOLLOWER
I20260620 18:44:08.724937 16380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3ede126e1342bba387386cee8c13c8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35403 } }
I20260620 18:44:08.725168 16380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:08.725317 16380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:08.725474 16380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:08.728039 16380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3ede126e1342bba387386cee8c13c8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35403 } }
I20260620 18:44:08.728464 16380 leader_election.cc:302] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e3ede126e1342bba387386cee8c13c8; no voters:
I20260620 18:44:08.728983 16380 leader_election.cc:288] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:08.729146 16385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:08.729787 16385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [term 1 LEADER]: Becoming Leader. State: Replica: 8e3ede126e1342bba387386cee8c13c8, State: Running, Role: LEADER
I20260620 18:44:08.730325 16385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3ede126e1342bba387386cee8c13c8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35403 } }
I20260620 18:44:08.731124 16380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:08.733372 16386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e3ede126e1342bba387386cee8c13c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3ede126e1342bba387386cee8c13c8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35403 } } }
I20260620 18:44:08.732981 16387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e3ede126e1342bba387386cee8c13c8. Latest consensus state: current_term: 1 leader_uuid: "8e3ede126e1342bba387386cee8c13c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3ede126e1342bba387386cee8c13c8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35403 } } }
I20260620 18:44:08.733816 16387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:08.733822 16386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:08.734858 16392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:08.741482 16392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:08.751205 16392 catalog_manager.cc:1347] Generated new cluster ID: 5805668c69a94c53a665c1d9fb0a9bd9
I20260620 18:44:08.751394 16392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:08.762369 16392 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:08.764233 16392 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:08.775221 16392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e3ede126e1342bba387386cee8c13c8: Generated new TSK 0
I20260620 18:44:08.776101 16392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:08.825419 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35403
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:09.159477 16406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:09.159761 16406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:09.159924 16406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:09.167135 16406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:09.167325 16406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:09.167553 16406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:09.177978 16406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35403
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:09.179430 16406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:09.180788 16406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:09.193979 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:09.194036 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:09.194006 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:09.194591 16406 server_base.cc:1034] running on GCE node
I20260620 18:44:09.195386 16406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:09.196317 16406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:09.197549 16406 hybrid_clock.cc:648] HybridClock initialized: now 1781981049197508 us; error 60 us; skew 500 ppm
I20260620 18:44:09.197887 16406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:09 dist-test-slave-zfp8 krb5kdc[16300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981049, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:09.201103 16406 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:09.202641 16406 webserver.cc:460] Webserver started at http://127.15.102.193:42557/ using document root <none> and password file <none>
I20260620 18:44:09.203398 16406 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:09.203543 16406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:09.203859 16406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:09.206692 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "2948c82a2bf54126a5137961136fed73"
format_stamp: "Formatted at 2026-06-20 18:44:09 on dist-test-slave-zfp8"
server_key: "01b46fa16b1748eb8a9c0e21bc20b10b"
server_key_iv: "dd7e81d7324994ae834701a34192fb12"
server_key_version: "encryptionkey@0"
I20260620 18:44:09.207547 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "2948c82a2bf54126a5137961136fed73"
format_stamp: "Formatted at 2026-06-20 18:44:09 on dist-test-slave-zfp8"
server_key: "01b46fa16b1748eb8a9c0e21bc20b10b"
server_key_iv: "dd7e81d7324994ae834701a34192fb12"
server_key_version: "encryptionkey@0"
I20260620 18:44:09.212453 16406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:44:09.216041 16421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:09.217394 16406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:09.217519 16406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "2948c82a2bf54126a5137961136fed73"
format_stamp: "Formatted at 2026-06-20 18:44:09 on dist-test-slave-zfp8"
server_key: "01b46fa16b1748eb8a9c0e21bc20b10b"
server_key_iv: "dd7e81d7324994ae834701a34192fb12"
server_key_version: "encryptionkey@0"
I20260620 18:44:09.217669 16406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:09.243395 16406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:09.247807 16406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:09.248047 16406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:09.248937 16406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:09.250339 16406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:09.250415 16406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:09.250499 16406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:09.250540 16406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:09.283522 16406 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:36467
I20260620 18:44:09.283614 16533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:36467 every 8 connection(s)
I20260620 18:44:09.284847 16406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:09.294485 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 16406
I20260620 18:44:09.294715 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:09.295470 15771 external_mini_cluster.cc:1428] Setting key 2b9e458b413d62c1a0b6240b960a9b21
Jun 20 18:44:09 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981049, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:09.305320 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:09.287406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:34487 (local address 127.15.102.254:35403)
0620 18:44:09.287848 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:09.287853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:09.288869 (+ 1016us) server_negotiation.cc:407] Connection header received
0620 18:44:09.290701 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:09.290706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:09.290807 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:09.290931 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:09.292476 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:09.293203 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:09.295291 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:09.295750 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:09.295883 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:09.299768 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:09.299773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:09.299776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:09.299806 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:09.302334 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:09.303040 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:09.303052 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:09.303055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:09.303165 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:09.303608 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:09.303613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:09.303616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:09.303875 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:09.304032 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:09.304739 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 18:44:09.304951 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
I20260620 18:44:09.306291 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:09.288016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35403 (local address 127.15.102.193:34487)
0620 18:44:09.288702 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:09.288749 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:09.290421 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:09.291171 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:09.291179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:09.291737 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:09.292301 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:09.292317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:09.293368 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:09.293373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:09.295078 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:09.295094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:09.296008 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:09.296018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:09.296266 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:09.297008 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:09.297030 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:09.299537 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:09.302477 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:09.302486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:09.302503 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:09.302911 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:09.303291 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:09.303297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:09.303301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:09.303488 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:09.304011 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:09.304019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:09.304353 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 18:44:09.304620 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 18:44:09.305138 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":268,"threads_started":1}
I20260620 18:44:09.307910 16534 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35403
I20260620 18:44:09.308214 16534 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:09.308928 16534 heartbeater.cc:511] Master 127.15.102.254:35403 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:44:09.310984 16345 ts_manager.cc:194] Registered new tserver with Master: 2948c82a2bf54126a5137961136fed73 (127.15.102.193:36467)
I20260620 18:44:09.312846 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:34487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:09.334967 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35403
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:09.654634 16540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:09.654928 16540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:09.655022 16540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:09.662235 16540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:09.662346 16540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:09.662462 16540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:09.672266 16540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35403
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:09.673487 16540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:09.674757 16540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:09.687652 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:09.687717 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:09.688297 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:09.688498 16540 server_base.cc:1034] running on GCE node
I20260620 18:44:09.689188 16540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:09.690115 16540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:09.691349 16540 hybrid_clock.cc:648] HybridClock initialized: now 1781981049691295 us; error 36 us; skew 500 ppm
I20260620 18:44:09.691697 16540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:09 dist-test-slave-zfp8 krb5kdc[16300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981049, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:09.694949 16540 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:09.696568 16540 webserver.cc:460] Webserver started at http://127.15.102.194:46501/ using document root <none> and password file <none>
I20260620 18:44:09.697273 16540 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:09.697415 16540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:09.697746 16540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:09.700781 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "c6aa5cfb99e84ebabc758a34c670777d"
format_stamp: "Formatted at 2026-06-20 18:44:09 on dist-test-slave-zfp8"
server_key: "c329fe2021af8852f8867de1d33abe71"
server_key_iv: "d65452bbb757a7f276d7b3925f25b016"
server_key_version: "encryptionkey@0"
I20260620 18:44:09.701611 16540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "c6aa5cfb99e84ebabc758a34c670777d"
format_stamp: "Formatted at 2026-06-20 18:44:09 on dist-test-slave-zfp8"
server_key: "c329fe2021af8852f8867de1d33abe71"
server_key_iv: "d65452bbb757a7f276d7b3925f25b016"
server_key_version: "encryptionkey@0"
I20260620 18:44:09.706867 16540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:44:09.710465 16555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:09.711930 16540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:44:09.712110 16540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "c6aa5cfb99e84ebabc758a34c670777d"
format_stamp: "Formatted at 2026-06-20 18:44:09 on dist-test-slave-zfp8"
server_key: "c329fe2021af8852f8867de1d33abe71"
server_key_iv: "d65452bbb757a7f276d7b3925f25b016"
server_key_version: "encryptionkey@0"
I20260620 18:44:09.712322 16540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:09.725926 16540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:09.730259 16540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:09.730549 16540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:09.731613 16540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:09.732965 16540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:09.733063 16540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:09.733209 16540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:09.733283 16540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:09.765492 16540 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:37783
I20260620 18:44:09.765535 16667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:37783 every 8 connection(s)
I20260620 18:44:09.766855 16540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:09.772014 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 16540
I20260620 18:44:09.772205 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:09.772764 15771 external_mini_cluster.cc:1428] Setting key e903d40a0b85a278d2ac57cbf910945b
Jun 20 18:44:09 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981049, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:09.790568 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:09.769428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:40107 (local address 127.15.102.254:35403)
0620 18:44:09.769647 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:09.769654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:09.770743 (+ 1089us) server_negotiation.cc:407] Connection header received
0620 18:44:09.772987 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:09.772991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:09.773060 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:09.773149 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:09.774947 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:09.775730 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:09.778370 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:09.778835 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:09.778950 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:09.782898 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:09.782908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:09.782917 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:09.782968 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:09.786827 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:09.787884 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:09.787889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:09.787891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:09.788029 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:09.788582 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:09.788585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:09.788587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:09.788979 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:09.789146 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:09.790125 (+ 979us) server_negotiation.cc:299] Negotiation successful
0620 18:44:09.790304 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
I20260620 18:44:09.792030 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:09.769907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35403 (local address 127.15.102.194:40107)
0620 18:44:09.770576 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:09.770616 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:09.772692 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:09.773438 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:09.773446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:09.774072 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:09.774768 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:09.774795 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:09.775909 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:09.775918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:09.778163 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:09.778180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:09.779078 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:09.779088 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:09.779456 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:09.780194 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:09.780225 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:09.782609 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:09.786991 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:09.786999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:09.787014 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:09.787738 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:09.788156 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:09.788164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:09.788171 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:09.788455 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:09.789137 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:09.789146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:09.789662 (+ 516us) client_negotiation.cc:769] Sending connection context
0620 18:44:09.790000 (+ 338us) client_negotiation.cc:240] Negotiation successful
0620 18:44:09.790615 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":265,"threads_started":1}
I20260620 18:44:09.793545 16668 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35403
I20260620 18:44:09.793913 16668 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
I20260620 18:44:09.794672 16668 heartbeater.cc:511] Master 127.15.102.254:35403 requested a full tablet report, sending...
I20260620 18:44:09.796223 16345 ts_manager.cc:194] Registered new tserver with Master: c6aa5cfb99e84ebabc758a34c670777d (127.15.102.194:37783)
I20260620 18:44:09.797019 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:40107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:09.814287 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35403
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:35493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:10.133268 16673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:10.133560 16673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:10.133656 16673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:10.140849 16673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:10.140957 16673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:10.141072 16673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:10.150888 16673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:35493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35403
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:10.152151 16673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:10.153381 16673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:10.166072 16678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:10.166224 16679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:10.166174 16681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:10.166867 16673 server_base.cc:1034] running on GCE node
I20260620 18:44:10.167495 16673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:10.168568 16673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:10.169790 16673 hybrid_clock.cc:648] HybridClock initialized: now 1781981050169756 us; error 52 us; skew 500 ppm
I20260620 18:44:10.170104 16673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981050, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:10.177855 16673 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:10.179404 16673 webserver.cc:460] Webserver started at http://127.15.102.195:46337/ using document root <none> and password file <none>
I20260620 18:44:10.180088 16673 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:10.180269 16673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:10.180558 16673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:10.183595 16673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "f81c0d6811d04c4795f5ec974425fd94"
format_stamp: "Formatted at 2026-06-20 18:44:10 on dist-test-slave-zfp8"
server_key: "0ae8a3b99dd101f0b90f6821a71c3050"
server_key_iv: "958753213e609ae33add2cd2d6207dab"
server_key_version: "encryptionkey@0"
I20260620 18:44:10.184383 16673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "f81c0d6811d04c4795f5ec974425fd94"
format_stamp: "Formatted at 2026-06-20 18:44:10 on dist-test-slave-zfp8"
server_key: "0ae8a3b99dd101f0b90f6821a71c3050"
server_key_iv: "958753213e609ae33add2cd2d6207dab"
server_key_version: "encryptionkey@0"
I20260620 18:44:10.189755 16673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:10.193259 16688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:10.194610 16673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:10.194799 16673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "f81c0d6811d04c4795f5ec974425fd94"
format_stamp: "Formatted at 2026-06-20 18:44:10 on dist-test-slave-zfp8"
server_key: "0ae8a3b99dd101f0b90f6821a71c3050"
server_key_iv: "958753213e609ae33add2cd2d6207dab"
server_key_version: "encryptionkey@0"
I20260620 18:44:10.195006 16673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:10.220850 16673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:10.226552 16673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:10.226943 16673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:10.228266 16673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:10.230346 16673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:10.230443 16673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:10.230568 16673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:10.230633 16673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:10.268877 16673 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:38231
I20260620 18:44:10.268999 16800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:38231 every 8 connection(s)
I20260620 18:44:10.270179 16673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:10.272192 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 16673
I20260620 18:44:10.272390 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:10.272812 15771 external_mini_cluster.cc:1428] Setting key 20c28993b7fb2bda9325420b8d361a7a
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981050, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:10.291479 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.273161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:37199 (local address 127.15.102.254:35403)
0620 18:44:10.273372 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.273379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.274876 (+ 1497us) server_negotiation.cc:407] Connection header received
0620 18:44:10.277031 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.277036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.277118 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.277242 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:10.278796 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.279631 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.281140 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.281565 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.281740 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.285223 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:10.285228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:10.285232 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:10.285263 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:10.288492 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.289371 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.289376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.289378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.289507 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.289877 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.289880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.289882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.290132 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:10.290249 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.290917 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.291102 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:44:10.292325 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.273794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35403 (local address 127.15.102.195:37199)
0620 18:44:10.274714 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:10.274761 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:10.276741 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:10.277483 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:10.277491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:10.278045 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:10.278631 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.278649 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.279783 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.279793 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.280996 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.281007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.281828 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.281838 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.282070 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.282746 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:10.282767 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:10.285000 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:10.288625 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:10.288631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:10.288643 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:10.289245 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:10.289607 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:10.289610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:10.289612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:10.289785 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:10.290251 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:10.290257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:10.290549 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 18:44:10.290776 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 18:44:10.291150 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":368,"threads_started":1}
I20260620 18:44:10.293771 16801 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35403
I20260620 18:44:10.294037 16801 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:10.294682 16801 heartbeater.cc:511] Master 127.15.102.254:35403 requested a full tablet report, sending...
I20260620 18:44:10.296098 16345 ts_manager.cc:194] Registered new tserver with Master: f81c0d6811d04c4795f5ec974425fd94 (127.15.102.195:38231)
I20260620 18:44:10.296973 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:37199
I20260620 18:44:10.301239 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:44:10.315923 16534 heartbeater.cc:503] Master 127.15.102.254:35403 was elected leader, sending a full tablet report...
I20260620 18:44:10.320580 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.307267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37176 (local address 127.15.102.254:35403)
0620 18:44:10.307453 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.307457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.307970 (+ 513us) server_negotiation.cc:407] Connection header received
0620 18:44:10.308104 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.308108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.308179 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.308273 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:10.310071 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.310806 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.312095 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.312529 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.312640 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.314872 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:10.314882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:10.314890 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:10.314935 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:10.318196 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.318781 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.318784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.318787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.318897 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.319225 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.319381 (+ 156us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.319385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.319698 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:10.319875 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.320148 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.320292 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:44:10.329905 16345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:10.336596 16345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:10.376549 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.348889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:37783 (local address 127.0.0.1:39016)
0620 18:44:10.362892 (+ 14003us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:10.362944 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:10.363196 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:10.365909 (+ 2713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:10.365921 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:10.365960 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:10.366387 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.366403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.369376 (+ 2973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.369385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.373330 (+ 3945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.373346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.375530 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.375538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.375737 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.375796 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:44:10.375933 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:44:10.376137 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13691,"mutex_wait_us":1,"thread_start_us":23214,"threads_started":1}
I20260620 18:44:10.377362 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.349026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.15.102.194:37783)
0620 18:44:10.362268 (+ 13242us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.362282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.364947 (+ 2665us) server_negotiation.cc:407] Connection header received
0620 18:44:10.365064 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.365071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.365389 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.365604 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:10.366605 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.369184 (+ 2579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.373550 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.374994 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.375162 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.375367 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.376819 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.376995 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13013,"thread_start_us":12860,"threads_started":1}
I20260620 18:44:10.379699 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.351426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36467 (local address 127.0.0.1:46770)
0620 18:44:10.372351 (+ 20925us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:10.372364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:10.372455 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:10.373022 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:10.373027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:10.373054 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:10.373498 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.373506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.374780 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.374788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.376968 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.376985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.378848 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.378855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.379028 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.379059 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:44:10.379156 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:44:10.379323 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20822,"thread_start_us":18817,"threads_started":1}
I20260620 18:44:10.380503 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.351876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.15.102.193:36467)
0620 18:44:10.369350 (+ 17474us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.369364 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.372478 (+ 3114us) server_negotiation.cc:407] Connection header received
0620 18:44:10.372574 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.372581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.372815 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.373010 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:10.373646 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.374631 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.377142 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.378629 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.378836 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.379002 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.379213 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.379411 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17218,"thread_start_us":17481,"threads_started":1}
I20260620 18:44:10.380949 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.349034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:38231 (local address 127.0.0.1:58414)
0620 18:44:10.370611 (+ 21577us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:10.370638 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:10.370795 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:10.372617 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:10.372622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:10.372648 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:10.373135 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.373149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.375338 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.375343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.377572 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.377586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.380208 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.380216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.380376 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.380406 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:44:10.380490 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 18:44:10.380639 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21396,"thread_start_us":14397,"threads_started":1}
I20260620 18:44:10.381652 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.349431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.15.102.195:38231)
0620 18:44:10.371429 (+ 21998us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.371438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.371472 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:44:10.371565 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.371572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.371830 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.372638 (+ 808us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:10.373399 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.374982 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.377716 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.379170 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.379419 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.379603 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.381139 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.381324 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21812,"thread_start_us":21631,"threads_started":1}
I20260620 18:44:10.381397 16603 tablet_service.cc:1467] Processing CreateTablet for tablet e661968f073444478765f59c13411c69 (DEFAULT_TABLE table=test-table [id=d0caaad7309f42c09b807026d0aca4b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:10.382957 16603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e661968f073444478765f59c13411c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:10.384486 16469 tablet_service.cc:1467] Processing CreateTablet for tablet e661968f073444478765f59c13411c69 (DEFAULT_TABLE table=test-table [id=d0caaad7309f42c09b807026d0aca4b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:10.385121 16736 tablet_service.cc:1467] Processing CreateTablet for tablet e661968f073444478765f59c13411c69 (DEFAULT_TABLE table=test-table [id=d0caaad7309f42c09b807026d0aca4b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:10.385622 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e661968f073444478765f59c13411c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:10.386420 16736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e661968f073444478765f59c13411c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:10.397172 16819 tablet_bootstrap.cc:492] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73: Bootstrap starting.
I20260620 18:44:10.399839 16820 tablet_bootstrap.cc:492] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d: Bootstrap starting.
I20260620 18:44:10.401608 16821 tablet_bootstrap.cc:492] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94: Bootstrap starting.
I20260620 18:44:10.401684 16819 tablet_bootstrap.cc:654] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:10.403715 16820 tablet_bootstrap.cc:654] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:10.403733 16819 log.cc:824] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:10.404894 16820 log.cc:824] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:10.405447 16821 tablet_bootstrap.cc:654] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:10.406440 16821 log.cc:824] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:10.407912 16819 tablet_bootstrap.cc:492] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73: No bootstrap required, opened a new log
I20260620 18:44:10.408215 16819 ts_tablet_manager.cc:1397] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260620 18:44:10.409432 16820 tablet_bootstrap.cc:492] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d: No bootstrap required, opened a new log
I20260620 18:44:10.409605 16821 tablet_bootstrap.cc:492] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94: No bootstrap required, opened a new log
I20260620 18:44:10.409731 16820 ts_tablet_manager.cc:1397] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:44:10.409864 16821 ts_tablet_manager.cc:1397] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:44:10.417770 16820 raft_consensus.cc:348] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.418185 16820 raft_consensus.cc:374] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:10.418298 16820 raft_consensus.cc:729] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6aa5cfb99e84ebabc758a34c670777d, State: Initialized, Role: FOLLOWER
I20260620 18:44:10.418963 16820 consensus_queue.cc:260] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.420181 16668 heartbeater.cc:503] Master 127.15.102.254:35403 was elected leader, sending a full tablet report...
I20260620 18:44:10.420800 16820 ts_tablet_manager.cc:1428] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260620 18:44:10.421147 16819 raft_consensus.cc:348] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.421574 16819 raft_consensus.cc:374] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:10.422092 16819 raft_consensus.cc:729] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2948c82a2bf54126a5137961136fed73, State: Initialized, Role: FOLLOWER
I20260620 18:44:10.422685 16819 consensus_queue.cc:260] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.423097 16821 raft_consensus.cc:348] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.423631 16821 raft_consensus.cc:374] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:10.423764 16821 raft_consensus.cc:729] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81c0d6811d04c4795f5ec974425fd94, State: Initialized, Role: FOLLOWER
I20260620 18:44:10.424620 16819 ts_tablet_manager.cc:1428] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260620 18:44:10.424496 16821 consensus_queue.cc:260] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.427197 16801 heartbeater.cc:503] Master 127.15.102.254:35403 was elected leader, sending a full tablet report...
I20260620 18:44:10.427894 16821 ts_tablet_manager.cc:1428] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20260620 18:44:10.518853 16669 tablet.cc:2369] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:44:10.521970 16802 tablet.cc:2369] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:44:10.537217 16535 tablet.cc:2369] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:10.556318 16826 raft_consensus.cc:482] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:10.556584 16826 raft_consensus.cc:504] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.558251 16826 leader_election.cc:288] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6aa5cfb99e84ebabc758a34c670777d (127.15.102.194:37783), f81c0d6811d04c4795f5ec974425fd94 (127.15.102.195:38231)
I20260620 18:44:10.563580 16825 raft_consensus.cc:482] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:10.563915 16825 raft_consensus.cc:504] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.565918 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.558672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:57773 (local address 127.15.102.194:37783)
0620 18:44:10.558912 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.558921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.559683 (+ 762us) server_negotiation.cc:407] Connection header received
0620 18:44:10.559976 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.559981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.560060 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.560126 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:10.561105 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.561852 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.563983 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.564917 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.565107 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.565191 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.565540 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.565665 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 18:44:10.566046 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.558421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:37783 (local address 127.15.102.193:57773)
0620 18:44:10.559573 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:10.559592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:10.559779 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:10.560171 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:10.560178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:10.560200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:10.560946 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.560954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.561979 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.561985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.563748 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.563762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.565127 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.565133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.565324 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.565370 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:44:10.565508 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:44:10.565649 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":1,"thread_start_us":1059,"threads_started":1}
I20260620 18:44:10.566010 16825 leader_election.cc:288] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f81c0d6811d04c4795f5ec974425fd94 (127.15.102.195:38231), 2948c82a2bf54126a5137961136fed73 (127.15.102.193:36467)
I20260620 18:44:10.566152 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.558877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:57881 (local address 127.15.102.195:38231)
0620 18:44:10.559103 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.559111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.559858 (+ 747us) server_negotiation.cc:407] Connection header received
0620 18:44:10.560026 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.560030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.560086 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.560191 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:10.561222 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.562063 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.564087 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.565227 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.565379 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.565446 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.565771 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.565902 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:44:10.566866 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.558747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:38231 (local address 127.15.102.193:57881)
0620 18:44:10.559756 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:10.559769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:10.559902 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:10.560234 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:10.560239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:10.560269 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:10.561051 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.561061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.562261 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.562265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.563857 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.563869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.565431 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.565440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.565599 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.565628 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:44:10.566357 (+ 729us) client_negotiation.cc:240] Negotiation successful
0620 18:44:10.566488 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":379,"threads_started":1}
I20260620 18:44:10.568261 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e661968f073444478765f59c13411c69" candidate_uuid: "2948c82a2bf54126a5137961136fed73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81c0d6811d04c4795f5ec974425fd94" is_pre_election: true
I20260620 18:44:10.568837 16756 raft_consensus.cc:2454] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2948c82a2bf54126a5137961136fed73 in term 0.
I20260620 18:44:10.569597 16422 leader_election.cc:302] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2948c82a2bf54126a5137961136fed73, f81c0d6811d04c4795f5ec974425fd94; no voters:
I20260620 18:44:10.568467 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e661968f073444478765f59c13411c69" candidate_uuid: "2948c82a2bf54126a5137961136fed73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6aa5cfb99e84ebabc758a34c670777d" is_pre_election: true
I20260620 18:44:10.570007 16826 raft_consensus.cc:2781] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:10.570160 16826 raft_consensus.cc:482] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:10.570225 16623 raft_consensus.cc:2454] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2948c82a2bf54126a5137961136fed73 in term 0.
I20260620 18:44:10.570284 16826 raft_consensus.cc:3037] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:10.573585 16826 raft_consensus.cc:504] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.574730 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e661968f073444478765f59c13411c69" candidate_uuid: "2948c82a2bf54126a5137961136fed73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6aa5cfb99e84ebabc758a34c670777d"
I20260620 18:44:10.574958 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.566953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:33187 (local address 127.15.102.193:36467)
0620 18:44:10.567174 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.567183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.568144 (+ 961us) server_negotiation.cc:407] Connection header received
0620 18:44:10.568404 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.568409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.568482 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.568547 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:10.570023 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.571156 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.573036 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.574011 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.574181 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.574249 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.574588 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.574727 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:44:10.574945 16623 raft_consensus.cc:3037] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:10.575132 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.566729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36467 (local address 127.15.102.194:33187)
0620 18:44:10.568051 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:10.568073 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:10.568256 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:10.568741 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:10.568745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:10.569188 (+ 443us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:10.569870 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.569878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.571343 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.571347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.572866 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.572878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.574232 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.574241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.574416 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.574453 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 18:44:10.574544 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:44:10.574673 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":66,"spinlock_wait_cycles":35200,"thread_start_us":2092,"threads_started":1}
I20260620 18:44:10.575583 16826 leader_election.cc:288] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [CANDIDATE]: Term 1 election: Requested vote from peers c6aa5cfb99e84ebabc758a34c670777d (127.15.102.194:37783), f81c0d6811d04c4795f5ec974425fd94 (127.15.102.195:38231)
I20260620 18:44:10.576086 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.566819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:53355 (local address 127.15.102.195:38231)
0620 18:44:10.567190 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.567196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.568039 (+ 843us) server_negotiation.cc:407] Connection header received
0620 18:44:10.568312 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.568317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.568390 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.568464 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:10.570016 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.571125 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.573725 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.574851 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.575009 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.575076 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.575691 (+ 615us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.575827 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260620 18:44:10.576136 16756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e661968f073444478765f59c13411c69" candidate_uuid: "2948c82a2bf54126a5137961136fed73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81c0d6811d04c4795f5ec974425fd94"
I20260620 18:44:10.576323 16756 raft_consensus.cc:3037] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:10.577314 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.566438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:38231 (local address 127.15.102.194:53355)
0620 18:44:10.567948 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:10.567969 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:10.568145 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:10.568677 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:10.568682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:10.568714 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:10.569843 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.569861 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.571301 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.571306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.573539 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:10.573556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.575291 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.575300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:10.575496 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.575527 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:44:10.576265 (+ 738us) client_negotiation.cc:240] Negotiation successful
0620 18:44:10.576413 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"mutex_wait_us":27,"thread_start_us":1108,"threads_started":1}
I20260620 18:44:10.578018 16489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e661968f073444478765f59c13411c69" candidate_uuid: "c6aa5cfb99e84ebabc758a34c670777d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2948c82a2bf54126a5137961136fed73" is_pre_election: true
I20260620 18:44:10.578151 16755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e661968f073444478765f59c13411c69" candidate_uuid: "c6aa5cfb99e84ebabc758a34c670777d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f81c0d6811d04c4795f5ec974425fd94" is_pre_election: true
I20260620 18:44:10.578526 16489 raft_consensus.cc:2379] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6aa5cfb99e84ebabc758a34c670777d in current term 1: Already voted for candidate 2948c82a2bf54126a5137961136fed73 in this term.
I20260620 18:44:10.578547 16623 raft_consensus.cc:2454] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2948c82a2bf54126a5137961136fed73 in term 1.
I20260620 18:44:10.579424 16756 raft_consensus.cc:2454] T e661968f073444478765f59c13411c69 P f81c0d6811d04c4795f5ec974425fd94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2948c82a2bf54126a5137961136fed73 in term 1.
I20260620 18:44:10.579463 16423 leader_election.cc:302] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2948c82a2bf54126a5137961136fed73, c6aa5cfb99e84ebabc758a34c670777d; no voters:
I20260620 18:44:10.579813 16826 raft_consensus.cc:2781] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:10.580029 16559 leader_election.cc:302] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6aa5cfb99e84ebabc758a34c670777d; no voters: 2948c82a2bf54126a5137961136fed73, f81c0d6811d04c4795f5ec974425fd94
I20260620 18:44:10.580276 16826 raft_consensus.cc:686] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [term 1 LEADER]: Becoming Leader. State: Replica: 2948c82a2bf54126a5137961136fed73, State: Running, Role: LEADER
I20260620 18:44:10.580693 16825 raft_consensus.cc:2726] T e661968f073444478765f59c13411c69 P c6aa5cfb99e84ebabc758a34c670777d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 18:44:10.580746 16826 consensus_queue.cc:237] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } }
I20260620 18:44:10.585332 16344 catalog_manager.cc:5495] T e661968f073444478765f59c13411c69 P 2948c82a2bf54126a5137961136fed73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2948c82a2bf54126a5137961136fed73 (127.15.102.193). New cstate: current_term: 1 leader_uuid: "2948c82a2bf54126a5137961136fed73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6aa5cfb99e84ebabc758a34c670777d" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 37783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f81c0d6811d04c4795f5ec974425fd94" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2948c82a2bf54126a5137961136fed73" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36467 } health_report { overall_health: HEALTHY } } }
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:10.617406 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.602525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.15.102.193:36467)
0620 18:44:10.602706 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.602712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.603725 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 18:44:10.603859 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.603863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.603944 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.604073 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:10.605500 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.606260 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.607498 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.607914 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.608009 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.611905 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:10.611918 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:10.611934 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:10.611975 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:10.614681 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.615394 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.615405 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.615407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.615551 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.615878 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.615881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.615883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.616272 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:10.616514 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.616852 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.617069 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981050, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:10.663198 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.649542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.15.102.193:36467)
0620 18:44:10.649801 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.649812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.650143 (+ 331us) server_negotiation.cc:407] Connection header received
0620 18:44:10.650292 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.650296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.650362 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.650451 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:10.651854 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.652542 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.653840 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.654231 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.654321 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.658193 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:10.658197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:10.658201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:10.658234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:10.660855 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.661437 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.661441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.661443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.661557 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.662011 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.662014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.662016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.662262 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:10.662454 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.662749 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.662905 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:10.709576 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.695087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.15.102.193:36467)
0620 18:44:10.695344 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.695353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.695741 (+ 388us) server_negotiation.cc:407] Connection header received
0620 18:44:10.695952 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.695963 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.696027 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.696137 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:10.697494 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.698350 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.699690 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.700135 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.700281 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.704495 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:10.704501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:10.704504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:10.704538 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:10.706922 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.707658 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.707664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.707667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.707788 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.708161 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.708166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.708169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.708469 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:10.708722 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.709029 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.709252 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 18:44:10.711287 16449 tablet_service.cc:2908] Rejecting scan request for tablet e661968f073444478765f59c13411c69: Uninitialized: clean time has not yet been initialized
I20260620 18:44:10.714394 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16406
I20260620 18:44:10.728317 16818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.715228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.15.102.193:36467)
0620 18:44:10.715390 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.715394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.715896 (+ 502us) server_negotiation.cc:407] Connection header received
0620 18:44:10.716124 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.716132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.716239 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.716370 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:10.717654 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.718441 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.719662 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.720096 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.720213 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.723071 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:10.723076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:10.723079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:10.723108 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:10.725872 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.726508 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.726512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.726514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.726668 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.726979 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.726982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.726984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.727372 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:10.727661 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.727874 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.728054 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 18:44:10.729030 16529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:10.831014 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 16406
I20260620 18:44:10.848371 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16540
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:44:10.863812 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.849152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.15.102.194:37783)
0620 18:44:10.849373 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.849379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.849449 (+ 70us) server_negotiation.cc:407] Connection header received
0620 18:44:10.849669 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.849680 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.849745 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.849872 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:10.851046 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.851856 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.853125 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.853524 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.853661 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.857621 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:10.857633 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:10.857649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:10.857684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:10.860644 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.861377 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.861387 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.861389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.861528 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.861948 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.861955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.861959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.862362 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:10.862711 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.863026 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.863485 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:44:10.864499 16663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:10.954890 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 16540
I20260620 18:44:10.971637 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16673
Jun 20 18:44:10 dist-test-slave-zfp8 krb5kdc[16300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:44:10.986346 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:10.972395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.15.102.195:38231)
0620 18:44:10.972569 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:10.972576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:10.972686 (+ 110us) server_negotiation.cc:407] Connection header received
0620 18:44:10.972863 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:10.972868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:10.972930 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:10.973040 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:10.974265 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.974948 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.976350 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:10.976754 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:10.976898 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:10.981020 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:10.981035 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:10.981052 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:10.981089 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:10.983661 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.984330 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.984337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.984339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.984484 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:10.984862 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:10.984868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:10.984871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:10.985221 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:10.985513 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:10.985793 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:44:10.986031 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:44:10.987083 16796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:11.073688 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 16673
I20260620 18:44:11.090461 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16313
I20260620 18:44:11.091602 16375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:11.198252 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 16313
2026-06-20T18:44:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4187 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:11 dist-test-slave-zfp8 krb5kdc[16851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:11 dist-test-slave-zfp8 krb5kdc[16851](info): set up 2 sockets
Jun 20 18:44:11 dist-test-slave-zfp8 krb5kdc[16851](info): commencing operation
krb5kdc: starting...
W20260620 18:44:12.275200 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:12 dist-test-slave-zfp8 krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:12.451076 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40005
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:40005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:12.770931 16864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:12.771314 16864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:12.771482 16864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:12.779114 16864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:12.779322 16864 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:44:12.779417 16864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:12.779541 16864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:12.779626 16864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:12.779712 16864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:12.790176 16864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:40005
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:12.791836 16864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:12.793316 16864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:12.805508 16869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:12.805472 16870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:12.805657 16872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:12.806301 16864 server_base.cc:1034] running on GCE node
I20260620 18:44:12.807200 16864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:12.808795 16864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:12.810031 16864 hybrid_clock.cc:648] HybridClock initialized: now 1781981052809957 us; error 46 us; skew 500 ppm
I20260620 18:44:12.810393 16864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:12 dist-test-slave-zfp8 krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981052, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:12.813501 16864 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:12.814989 16864 webserver.cc:460] Webserver started at http://127.15.102.254:39159/ using document root <none> and password file <none>
I20260620 18:44:12.815774 16864 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:12.815910 16864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:12.816242 16864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:12.819293 16864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "c7e1665024374d17bb64c8924522d5cf"
format_stamp: "Formatted at 2026-06-20 18:44:12 on dist-test-slave-zfp8"
server_key: "8fed2ba579a44390ca892558a44d8eda"
server_key_iv: "8a96f7b01dbe90e8ce7895426947d035"
server_key_version: "encryptionkey@0"
I20260620 18:44:12.820106 16864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "c7e1665024374d17bb64c8924522d5cf"
format_stamp: "Formatted at 2026-06-20 18:44:12 on dist-test-slave-zfp8"
server_key: "8fed2ba579a44390ca892558a44d8eda"
server_key_iv: "8a96f7b01dbe90e8ce7895426947d035"
server_key_version: "encryptionkey@0"
I20260620 18:44:12.825357 16864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:44:12.829231 16879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:12.830669 16864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:12.830897 16864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "c7e1665024374d17bb64c8924522d5cf"
format_stamp: "Formatted at 2026-06-20 18:44:12 on dist-test-slave-zfp8"
server_key: "8fed2ba579a44390ca892558a44d8eda"
server_key_iv: "8a96f7b01dbe90e8ce7895426947d035"
server_key_version: "encryptionkey@0"
I20260620 18:44:12.831138 16864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:12.878463 16864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:12.883385 16864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:12.883715 16864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:12.905514 16864 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:40005
I20260620 18:44:12.905522 16940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:40005 every 8 connection(s)
I20260620 18:44:12.906931 16864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:12.910772 16941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:12.910997 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 16864
I20260620 18:44:12.911262 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:12.912068 15771 external_mini_cluster.cc:1428] Setting key a5c7018f538e69bae0a30f728e67a4f0
I20260620 18:44:12.924767 16941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf: Bootstrap starting.
Jun 20 18:44:12 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:12.928622 16941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:12.929833 16941 log.cc:824] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:12.932775 16941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf: No bootstrap required, opened a new log
I20260620 18:44:12.936124 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:12.914187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.15.102.254:40005)
0620 18:44:12.914979 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:12.914991 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:12.915032 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:44:12.917074 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:12.917098 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:12.917522 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:12.917857 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:12.919603 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:12.920989 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:12.922797 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:12.923536 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:12.923666 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:12.928436 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:12.928444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:12.928458 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:12.928494 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:12.931713 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:12.932474 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:12.932482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:12.932497 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:12.932682 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:12.933150 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:12.933154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:12.933156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:12.933652 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:12.933944 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:12.934463 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 18:44:12.934993 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":368,"threads_started":1}
I20260620 18:44:12.941174 16941 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e1665024374d17bb64c8924522d5cf" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40005 } }
I20260620 18:44:12.941540 16941 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:12.941645 16941 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e1665024374d17bb64c8924522d5cf, State: Initialized, Role: FOLLOWER
I20260620 18:44:12.942271 16941 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e1665024374d17bb64c8924522d5cf" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40005 } }
I20260620 18:44:12.942502 16941 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:12.942592 16941 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:12.942705 16941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:12.944819 16941 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e1665024374d17bb64c8924522d5cf" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40005 } }
I20260620 18:44:12.945250 16941 leader_election.cc:302] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7e1665024374d17bb64c8924522d5cf; no voters:
I20260620 18:44:12.945755 16941 leader_election.cc:288] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:12.945895 16946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:12.946956 16946 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [term 1 LEADER]: Becoming Leader. State: Replica: c7e1665024374d17bb64c8924522d5cf, State: Running, Role: LEADER
I20260620 18:44:12.947590 16946 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e1665024374d17bb64c8924522d5cf" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40005 } }
I20260620 18:44:12.948086 16941 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:12.950362 16947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7e1665024374d17bb64c8924522d5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e1665024374d17bb64c8924522d5cf" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40005 } } }
I20260620 18:44:12.950675 16947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:12.951555 16954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:12.951393 16948 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7e1665024374d17bb64c8924522d5cf. Latest consensus state: current_term: 1 leader_uuid: "c7e1665024374d17bb64c8924522d5cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e1665024374d17bb64c8924522d5cf" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40005 } } }
I20260620 18:44:12.951772 16948 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:12.957216 16954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:12.967356 16954 catalog_manager.cc:1347] Generated new cluster ID: 9848bbc350a340e3a4bb4f775204d180
I20260620 18:44:12.967478 16954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:12.979487 16954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:12.980783 16954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:12.992071 16954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf: Generated new TSK 0
I20260620 18:44:12.993139 16954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:13.043462 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40005
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:13.381325 16967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:13.381609 16967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:13.381704 16967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:13.388912 16967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:13.389024 16967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:13.389106 16967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:44:13.389197 16967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:13.398775 16967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:40005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:13.400003 16967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:13.401513 16967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:13.414417 16972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:13.414397 16975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:13.414594 16973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:13.415266 16967 server_base.cc:1034] running on GCE node
I20260620 18:44:13.415917 16967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:13.416823 16967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:13.418071 16967 hybrid_clock.cc:648] HybridClock initialized: now 1781981053418002 us; error 48 us; skew 500 ppm
I20260620 18:44:13.418375 16967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:13 dist-test-slave-zfp8 krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981053, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:13.421563 16967 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:13.423048 16967 webserver.cc:460] Webserver started at http://127.15.102.193:38369/ using document root <none> and password file <none>
I20260620 18:44:13.423796 16967 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:13.423923 16967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:13.424235 16967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:13.427001 16967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "170daff68d0a4494b6d2d7e9757ada3d"
format_stamp: "Formatted at 2026-06-20 18:44:13 on dist-test-slave-zfp8"
server_key: "ec9dc59b59ebdf0d5518c3c416aff059"
server_key_iv: "b58181faa2447ac7ad24927b0fce24f2"
server_key_version: "encryptionkey@0"
I20260620 18:44:13.427834 16967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "170daff68d0a4494b6d2d7e9757ada3d"
format_stamp: "Formatted at 2026-06-20 18:44:13 on dist-test-slave-zfp8"
server_key: "ec9dc59b59ebdf0d5518c3c416aff059"
server_key_iv: "b58181faa2447ac7ad24927b0fce24f2"
server_key_version: "encryptionkey@0"
I20260620 18:44:13.432780 16967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:44:13.436316 16982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:13.437700 16967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:44:13.437885 16967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "170daff68d0a4494b6d2d7e9757ada3d"
format_stamp: "Formatted at 2026-06-20 18:44:13 on dist-test-slave-zfp8"
server_key: "ec9dc59b59ebdf0d5518c3c416aff059"
server_key_iv: "b58181faa2447ac7ad24927b0fce24f2"
server_key_version: "encryptionkey@0"
I20260620 18:44:13.438094 16967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:13.456423 16967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:13.460572 16967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:13.460848 16967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:13.463650 16967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:13.463764 16967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:13.463905 16967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:13.463981 16967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:44:13 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981053, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:13.487957 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:13.464252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:36617 (local address 127.15.102.254:40005)
0620 18:44:13.464745 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:13.464750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:13.466062 (+ 1312us) server_negotiation.cc:407] Connection header received
0620 18:44:13.468307 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:13.468315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:13.468408 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:13.468543 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:13.470983 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.472093 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.474760 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.475168 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.475376 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:13.481114 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:13.481124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:13.481130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:13.481179 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:13.484819 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:13.485580 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:13.485587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:13.485591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:13.485718 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:13.486184 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:13.486192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:13.486196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:13.486478 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:13.486626 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:13.487368 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 18:44:13.487590 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":277,"threads_started":1}
I20260620 18:44:13.489048 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:13.465015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40005 (local address 127.15.102.193:36617)
0620 18:44:13.465875 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:13.465928 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:13.467924 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:13.468855 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:13.468867 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:13.469765 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:13.470719 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:13.470750 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.472246 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.472252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:13.474595 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:13.474606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.475728 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.475747 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:13.475976 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:13.476881 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:13.476918 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:13.480746 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:13.484981 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:13.484990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:13.485008 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:13.485446 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:13.485841 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:13.485846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:13.485850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:13.486065 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:13.486615 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:13.486623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:13.486991 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 18:44:13.487293 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 18:44:13.487857 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":365,"threads_started":1}
I20260620 18:44:13.515610 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:13.496209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:48059 (local address 127.15.102.254:40005)
0620 18:44:13.496395 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:13.496402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:13.500250 (+ 3848us) server_negotiation.cc:407] Connection header received
0620 18:44:13.500481 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:13.500491 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:13.500592 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:13.500707 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:13.501768 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.503067 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.505432 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.506164 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.506305 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:13.509363 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:13.509372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:13.509379 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:13.509427 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:13.512223 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:13.513037 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:13.513044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:13.513049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:13.513225 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:13.513791 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:13.513797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:13.513801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:13.514107 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:13.514218 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:13.514884 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 18:44:13.515152 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:44:13.515885 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:13.496353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40005 (local address 127.15.102.193:48059)
0620 18:44:13.500097 (+ 3744us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:13.500119 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:13.500298 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:13.500767 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:13.500773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:13.501327 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:13.501607 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:13.501616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.503228 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.503273 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:13.505212 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:13.505231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.506587 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.506596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:13.506804 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:13.507640 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:13.507670 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:13.509104 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:13.512382 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:13.512389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:13.512395 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:13.512904 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:13.513413 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:13.513422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:13.513429 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:13.513634 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:13.514240 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:13.514248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:13.514521 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 18:44:13.514714 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 18:44:13.515212 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3595,"thread_start_us":3471,"threads_started":1}
I20260620 18:44:13.517819 17102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:37159 every 8 connection(s)
I20260620 18:44:13.517820 16967 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:37159
I20260620 18:44:13.519577 16967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:13.521672 17103 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40005
I20260620 18:44:13.521876 17103 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:13.522454 17103 heartbeater.cc:511] Master 127.15.102.254:40005 requested a full tablet report, sending...
I20260620 18:44:13.523921 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 16967
I20260620 18:44:13.524138 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:13.524500 16896 ts_manager.cc:194] Registered new tserver with Master: 170daff68d0a4494b6d2d7e9757ada3d (127.15.102.193:37159)
I20260620 18:44:13.524822 15771 external_mini_cluster.cc:1428] Setting key c6b7efb173c1f5277f32e9ee3c85da73
I20260620 18:44:13.526290 16896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:36617
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:13.564288 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40005
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:13.890192 17107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:13.890472 17107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:13.890641 17107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:13.897970 17107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:13.898087 17107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:13.898241 17107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:44:13.898360 17107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:13.908627 17107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:40005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:13.909934 17107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:13.911293 17107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:13.924891 17115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:13.925392 17112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:13.925442 17113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:13.928725 17107 server_base.cc:1034] running on GCE node
I20260620 18:44:13.929350 17107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:13.930351 17107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:13.931560 17107 hybrid_clock.cc:648] HybridClock initialized: now 1781981053931499 us; error 33 us; skew 500 ppm
I20260620 18:44:13.931826 17107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:13 dist-test-slave-zfp8 krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981053, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:13.934850 17107 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:13.936511 17107 webserver.cc:460] Webserver started at http://127.15.102.194:45173/ using document root <none> and password file <none>
I20260620 18:44:13.937379 17107 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:13.937515 17107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:13.937852 17107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:13.940979 17107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "684f77b470a04fc198d2761ae73d51ec"
format_stamp: "Formatted at 2026-06-20 18:44:13 on dist-test-slave-zfp8"
server_key: "5a363e4ac5336824e36fa328f5fd05e1"
server_key_iv: "ec12313cdc7fb17ca83959dadef81503"
server_key_version: "encryptionkey@0"
I20260620 18:44:13.941890 17107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "684f77b470a04fc198d2761ae73d51ec"
format_stamp: "Formatted at 2026-06-20 18:44:13 on dist-test-slave-zfp8"
server_key: "5a363e4ac5336824e36fa328f5fd05e1"
server_key_iv: "ec12313cdc7fb17ca83959dadef81503"
server_key_version: "encryptionkey@0"
I20260620 18:44:13.947188 17107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:44:13.950917 17122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:13.952322 17107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:13.952517 17107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "684f77b470a04fc198d2761ae73d51ec"
format_stamp: "Formatted at 2026-06-20 18:44:13 on dist-test-slave-zfp8"
server_key: "5a363e4ac5336824e36fa328f5fd05e1"
server_key_iv: "ec12313cdc7fb17ca83959dadef81503"
server_key_version: "encryptionkey@0"
I20260620 18:44:13.952750 17107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:13.976660 17107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:13.981066 17107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:13.981364 17107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:13.984279 17107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:13.984391 17107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:13.984544 17107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:13.984656 17107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 18:44:13 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981053, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:14.010141 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:13.984737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:47165 (local address 127.15.102.254:40005)
0620 18:44:13.984933 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:13.984942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:13.986525 (+ 1583us) server_negotiation.cc:407] Connection header received
0620 18:44:13.988854 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:13.988860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:13.988953 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:13.989076 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:13.991456 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.992623 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.995543 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.996144 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.996289 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.001677 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:14.001687 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:14.001693 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:14.001739 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:14.005856 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.006903 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.006909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.006912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.007091 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.007915 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.007922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.007926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.008247 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:14.008429 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.009551 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.009777 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:44:14.011428 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:13.985510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40005 (local address 127.15.102.194:47165)
0620 18:44:13.986335 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:13.986387 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:13.988508 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:13.989389 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:13.989404 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:13.990230 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:13.991177 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:13.991204 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.992779 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.992784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:13.995308 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:13.995326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:13.996411 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:13.996419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:13.996798 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:13.997672 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:13.997714 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:14.001318 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:14.006052 (+ 4734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:14.006065 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:14.006092 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:14.006748 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:14.007251 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:14.007259 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:14.007268 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:14.007745 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:14.008511 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:14.008520 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:14.009102 (+ 582us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.009446 (+ 344us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.010021 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":375,"threads_started":1}
I20260620 18:44:14.033211 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.017391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40005 (local address 127.15.102.194:34067)
0620 18:44:14.018120 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:14.018152 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:14.018326 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:14.018678 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:14.018686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:14.019487 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:14.020059 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.020078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.021610 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.021620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.023357 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.023377 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.024600 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.024605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.024714 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.025449 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:14.025466 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:14.026228 (+ 762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:14.029933 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:14.029942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:14.029949 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:14.030654 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:14.031048 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:14.031056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:14.031062 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:14.031271 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:14.031954 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:14.031959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:14.032240 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.032437 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.032876 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":357,"threads_started":1}
I20260620 18:44:14.033555 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.017421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:34067 (local address 127.15.102.254:40005)
0620 18:44:14.017650 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.017659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.018255 (+ 596us) server_negotiation.cc:407] Connection header received
0620 18:44:14.018471 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.018475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.018534 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.018612 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:14.020236 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.021423 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.023529 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.024413 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.024580 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.026458 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:14.026468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:14.026476 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:14.026527 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:14.029793 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.030781 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.030784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.030786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.030934 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.031420 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.031427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.031432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.031809 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:14.031972 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.032881 (+ 909us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.033182 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:44:14.041720 17107 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:36673
I20260620 18:44:14.041739 17241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:36673 every 8 connection(s)
I20260620 18:44:14.043015 17107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:14.045185 17242 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40005
I20260620 18:44:14.045374 17242 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:14.045941 17242 heartbeater.cc:511] Master 127.15.102.254:40005 requested a full tablet report, sending...
I20260620 18:44:14.047130 16896 ts_manager.cc:194] Registered new tserver with Master: 684f77b470a04fc198d2761ae73d51ec (127.15.102.194:36673)
I20260620 18:44:14.048102 16896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:47165
I20260620 18:44:14.052347 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 17107
I20260620 18:44:14.052564 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:14.052980 15771 external_mini_cluster.cc:1428] Setting key 701c1460ef19420ec9458902dfd72fcb
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:14.091501 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40005
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:14.417129 17246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:14.417433 17246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:14.417531 17246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:14.424757 17246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:14.424873 17246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:14.424957 17246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:44:14.425034 17246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:14.435004 17246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:40005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:14.436425 17246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:14.437726 17246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:14.451709 17252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:14.451817 17254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:14.452641 17251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:14.456983 17246 server_base.cc:1034] running on GCE node
I20260620 18:44:14.457759 17246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:14.459084 17246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:14.460345 17246 hybrid_clock.cc:648] HybridClock initialized: now 1781981054460248 us; error 62 us; skew 500 ppm
I20260620 18:44:14.460727 17246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:14 dist-test-slave-zfp8 krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981054, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:14.463997 17246 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:14.465674 17246 webserver.cc:460] Webserver started at http://127.15.102.195:35511/ using document root <none> and password file <none>
I20260620 18:44:14.466445 17246 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:14.466578 17246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:14.466913 17246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:14.469944 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6"
format_stamp: "Formatted at 2026-06-20 18:44:14 on dist-test-slave-zfp8"
server_key: "f19ce17b602fae6c5c8a088d162ccc84"
server_key_iv: "3adac59bc1864f5bd43e77697246a089"
server_key_version: "encryptionkey@0"
I20260620 18:44:14.470777 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6"
format_stamp: "Formatted at 2026-06-20 18:44:14 on dist-test-slave-zfp8"
server_key: "f19ce17b602fae6c5c8a088d162ccc84"
server_key_iv: "3adac59bc1864f5bd43e77697246a089"
server_key_version: "encryptionkey@0"
I20260620 18:44:14.475936 17246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:44:14.479406 17261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:14.480731 17246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:14.480909 17246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6"
format_stamp: "Formatted at 2026-06-20 18:44:14 on dist-test-slave-zfp8"
server_key: "f19ce17b602fae6c5c8a088d162ccc84"
server_key_iv: "3adac59bc1864f5bd43e77697246a089"
server_key_version: "encryptionkey@0"
I20260620 18:44:14.481159 17246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:14.510569 17246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:14.515676 17246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:14.515972 17246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:14.519225 17246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:14.519753 17246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20260620 18:44:14.519896 17246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:14.519961 17246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:14.529594 17103 heartbeater.cc:503] Master 127.15.102.254:40005 was elected leader, sending a full tablet report...
Jun 20 18:44:14 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981054, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:14.543598 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.519524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:42749 (local address 127.15.102.254:40005)
0620 18:44:14.519724 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.519730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.521436 (+ 1706us) server_negotiation.cc:407] Connection header received
0620 18:44:14.523529 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.523536 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.523621 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.523725 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:14.526354 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.527709 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.530768 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.531205 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.531388 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.536719 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:14.536724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:14.536728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:14.536794 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:14.540329 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.541161 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.541165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.541167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.541295 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.541747 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.541755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.541760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.542045 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:14.542198 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.542966 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.543190 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:44:14.544667 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.520295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40005 (local address 127.15.102.195:42749)
0620 18:44:14.521260 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:14.521308 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:14.523163 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:14.524134 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:14.524142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:14.524708 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:14.526091 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.526124 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.527918 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.527925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.530583 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.530595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.531510 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.531517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.531835 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.532684 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:14.532723 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:14.536447 (+ 3724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:14.540488 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:14.540501 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:14.540519 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:14.540964 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:14.541414 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:14.541418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:14.541424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:14.541632 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:14.542203 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:14.542213 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:14.542577 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.542860 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.543435 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":384,"threads_started":1}
I20260620 18:44:14.566947 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.550411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:43653 (local address 127.15.102.254:40005)
0620 18:44:14.550643 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.550652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.551289 (+ 637us) server_negotiation.cc:407] Connection header received
0620 18:44:14.551491 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.551499 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.551616 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.551766 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:14.552650 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.554092 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.556379 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.556887 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.557060 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.559731 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:14.559736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:14.559740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:14.559776 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:14.563537 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.564664 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.564671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.564675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.564818 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.565241 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.565248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.565255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.565667 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:14.565866 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.566301 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.566541 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:44:14.567049 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.550488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40005 (local address 127.15.102.195:43653)
0620 18:44:14.551155 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:14.551177 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:14.551366 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:14.551801 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:14.551805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:14.552214 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:14.552504 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.552513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.554270 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.554275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.556040 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.556053 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.557099 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.557106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.557378 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.558196 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:14.558216 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:14.559455 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:14.563696 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:14.563701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:14.563706 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:14.564509 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:14.564949 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:14.564954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:14.564958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:14.565103 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:14.565820 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:14.565825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:14.566028 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.566179 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.566559 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":363,"threads_started":1}
I20260620 18:44:14.581529 17246 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:45793
I20260620 18:44:14.581552 17380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:45793 every 8 connection(s)
I20260620 18:44:14.582764 17246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:14.585036 17381 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40005
I20260620 18:44:14.585381 17381 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:14.586143 17381 heartbeater.cc:511] Master 127.15.102.254:40005 requested a full tablet report, sending...
I20260620 18:44:14.587622 16896 ts_manager.cc:194] Registered new tserver with Master: a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793)
I20260620 18:44:14.588538 16896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:42749
I20260620 18:44:14.590255 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 17246
I20260620 18:44:14.590481 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:14.591071 15771 external_mini_cluster.cc:1428] Setting key dbb6cb514a05844676a022a73c06e6ae
I20260620 18:44:14.594360 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:44:14 dist-test-slave-zfp8 krb5kdc[16851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:44:14 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:14.643332 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.629055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.15.102.254:40005)
0620 18:44:14.629259 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.629264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.629572 (+ 308us) server_negotiation.cc:407] Connection header received
0620 18:44:14.629747 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.629752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.629821 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.629926 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:14.631311 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.632087 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.633360 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.633821 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.633931 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.637928 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:14.637933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:14.637936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:14.637970 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:14.640705 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.641440 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.641449 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.641451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.641577 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:14.641980 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:14.641983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:14.641985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:14.642236 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:14.642419 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.642828 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.642987 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:44:14.647743 16896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:14.655301 16896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:14.681442 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.668643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59512 (local address 127.15.102.195:45793)
0620 18:44:14.673412 (+ 4769us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.673426 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.673469 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:44:14.673596 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.673608 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.673923 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.674137 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:14.674974 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.676604 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.678823 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.680293 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.680480 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.680659 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.680942 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.681107 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4554,"thread_start_us":4331,"threads_started":1}
I20260620 18:44:14.681490 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.667917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:45793 (local address 127.0.0.1:59512)
0620 18:44:14.672614 (+ 4697us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:14.672637 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:14.672778 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:14.674256 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:14.674263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:14.674310 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:14.674743 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.674757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.676753 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.676759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.678652 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.678667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.680491 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.680497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.680691 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.680746 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.680879 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.681070 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4542,"mutex_wait_us":6,"thread_start_us":4335,"threads_started":1}
I20260620 18:44:14.682458 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.667684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.15.102.194:36673)
0620 18:44:14.674045 (+ 6361us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.674055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.674081 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:44:14.674169 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.674178 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.674405 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.674606 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:14.675987 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.677149 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.679890 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.681164 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.681327 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.681488 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.681990 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.682147 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6153,"thread_start_us":6016,"threads_started":1}
I20260620 18:44:14.682467 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.667510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:36673 (local address 127.0.0.1:45484)
0620 18:44:14.670197 (+ 2687us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:14.670257 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:14.670471 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:14.675102 (+ 4631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:14.675106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:14.675131 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:14.675791 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.675808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.677280 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.677291 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.679699 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.679713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.681632 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.681638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.681813 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.681851 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.681949 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.682083 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2337,"mutex_wait_us":1,"thread_start_us":2155,"threads_started":1}
I20260620 18:44:14.682843 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.673620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37159 (local address 127.0.0.1:51674)
0620 18:44:14.674284 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:14.674303 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:14.674519 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:14.675395 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:14.675401 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:14.675432 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:14.675875 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.675888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.677447 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.677451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.679991 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.680002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.682034 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.682041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.682222 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.682251 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.682358 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.682480 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":23,"thread_start_us":414,"threads_started":1}
I20260620 18:44:14.683291 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.673010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.15.102.193:37159)
0620 18:44:14.673572 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.673580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.674404 (+ 824us) server_negotiation.cc:407] Connection header received
0620 18:44:14.674750 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.674761 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.674961 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.675123 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:14.676092 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.677305 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.680146 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.681578 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.681744 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.681885 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.682832 (+ 947us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.683005 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":322,"threads_started":1}
I20260620 18:44:14.685724 17316 tablet_service.cc:1467] Processing CreateTablet for tablet 34677da537b443d1961f643dd4c43bb0 (DEFAULT_TABLE table=test-table [id=7142a450288a4b40a332d76e0d14a7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:14.685922 17172 tablet_service.cc:1467] Processing CreateTablet for tablet 34677da537b443d1961f643dd4c43bb0 (DEFAULT_TABLE table=test-table [id=7142a450288a4b40a332d76e0d14a7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:14.686896 17316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34677da537b443d1961f643dd4c43bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:14.687054 17172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34677da537b443d1961f643dd4c43bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:14.689898 17033 tablet_service.cc:1467] Processing CreateTablet for tablet 34677da537b443d1961f643dd4c43bb0 (DEFAULT_TABLE table=test-table [id=7142a450288a4b40a332d76e0d14a7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:14.691367 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34677da537b443d1961f643dd4c43bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:14.699047 17400 tablet_bootstrap.cc:492] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Bootstrap starting.
I20260620 18:44:14.701169 17401 tablet_bootstrap.cc:492] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Bootstrap starting.
I20260620 18:44:14.703147 17400 tablet_bootstrap.cc:654] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:14.704510 17400 log.cc:824] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:14.704676 17401 tablet_bootstrap.cc:654] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:14.704734 17402 tablet_bootstrap.cc:492] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Bootstrap starting.
I20260620 18:44:14.705958 17401 log.cc:824] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:14.708262 17400 tablet_bootstrap.cc:492] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: No bootstrap required, opened a new log
I20260620 18:44:14.708360 17402 tablet_bootstrap.cc:654] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:14.708560 17400 ts_tablet_manager.cc:1397] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:44:14.709568 17401 tablet_bootstrap.cc:492] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: No bootstrap required, opened a new log
I20260620 18:44:14.709604 17402 log.cc:824] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:14.709842 17401 ts_tablet_manager.cc:1397] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:44:14.712584 17402 tablet_bootstrap.cc:492] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: No bootstrap required, opened a new log
I20260620 18:44:14.712857 17402 ts_tablet_manager.cc:1397] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:44:14.718923 17401 raft_consensus.cc:348] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.719259 17400 raft_consensus.cc:348] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.719774 17401 raft_consensus.cc:374] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:14.719895 17401 raft_consensus.cc:729] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37b11cbe6cc4641a4d39ed15ca24eb6, State: Initialized, Role: FOLLOWER
I20260620 18:44:14.719977 17400 raft_consensus.cc:374] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:14.720105 17400 raft_consensus.cc:729] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684f77b470a04fc198d2761ae73d51ec, State: Initialized, Role: FOLLOWER
I20260620 18:44:14.720582 17401 consensus_queue.cc:260] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.720697 17400 consensus_queue.cc:260] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.721738 17381 heartbeater.cc:503] Master 127.15.102.254:40005 was elected leader, sending a full tablet report...
I20260620 18:44:14.722446 17401 ts_tablet_manager.cc:1428] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260620 18:44:14.723416 17242 heartbeater.cc:503] Master 127.15.102.254:40005 was elected leader, sending a full tablet report...
I20260620 18:44:14.723827 17402 raft_consensus.cc:348] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.724139 17400 ts_tablet_manager.cc:1428] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 18:44:14.724248 17402 raft_consensus.cc:374] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:14.724337 17402 raft_consensus.cc:729] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170daff68d0a4494b6d2d7e9757ada3d, State: Initialized, Role: FOLLOWER
I20260620 18:44:14.724922 17402 consensus_queue.cc:260] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.726681 17402 ts_tablet_manager.cc:1428] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260620 18:44:14.772025 17104 tablet.cc:2369] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:44:14.795050 17243 tablet.cc:2369] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:44:14.834429 17382 tablet.cc:2369] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:14.850781 17407 raft_consensus.cc:482] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:14.851140 17407 raft_consensus.cc:504] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.853323 17407 leader_election.cc:288] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793), 170daff68d0a4494b6d2d7e9757ada3d (127.15.102.193:37159)
I20260620 18:44:14.860889 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.853366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:43475 (local address 127.15.102.195:45793)
0620 18:44:14.853603 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.853607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.854239 (+ 632us) server_negotiation.cc:407] Connection header received
0620 18:44:14.854468 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.854477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.854569 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.854662 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:14.855486 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.856500 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.859054 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.859928 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.860049 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.860099 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.860494 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.860626 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:44:14.861027 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.853420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:45793 (local address 127.15.102.194:43475)
0620 18:44:14.854135 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:14.854151 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:14.854320 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:14.854805 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:14.854813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:14.854846 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:14.855343 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.855354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.856648 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.856655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.858893 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.858910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.860144 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.860151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.860324 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.860357 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.860451 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.860551 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":34,"spinlock_wait_cycles":19840,"thread_start_us":354,"threads_started":1}
I20260620 18:44:14.861047 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.854464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:40833 (local address 127.15.102.193:37159)
0620 18:44:14.854693 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.854698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.855039 (+ 341us) server_negotiation.cc:407] Connection header received
0620 18:44:14.855216 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.855221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.855332 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.855389 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:14.856197 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.857101 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.859328 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.860144 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.860305 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.860372 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.860691 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.860813 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:44:14.861230 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.854222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37159 (local address 127.15.102.194:40833)
0620 18:44:14.854929 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:14.854942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:14.855059 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:14.855589 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:14.855593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:14.855616 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:14.855980 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.855994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.857271 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.857275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.859147 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:14.859160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.860337 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.860348 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:14.860515 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.860547 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:44:14.860635 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:44:14.860756 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":23,"spinlock_wait_cycles":896,"thread_start_us":381,"threads_started":1}
I20260620 18:44:14.861969 17336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34677da537b443d1961f643dd4c43bb0" candidate_uuid: "684f77b470a04fc198d2761ae73d51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" is_pre_election: true
I20260620 18:44:14.862151 17058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34677da537b443d1961f643dd4c43bb0" candidate_uuid: "684f77b470a04fc198d2761ae73d51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170daff68d0a4494b6d2d7e9757ada3d" is_pre_election: true
I20260620 18:44:14.862497 17336 raft_consensus.cc:2454] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 684f77b470a04fc198d2761ae73d51ec in term 0.
I20260620 18:44:14.862695 17058 raft_consensus.cc:2454] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 684f77b470a04fc198d2761ae73d51ec in term 0.
I20260620 18:44:14.863154 17125 leader_election.cc:302] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684f77b470a04fc198d2761ae73d51ec, a37b11cbe6cc4641a4d39ed15ca24eb6; no voters:
I20260620 18:44:14.863554 17407 raft_consensus.cc:2781] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:14.863708 17407 raft_consensus.cc:482] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:14.863834 17407 raft_consensus.cc:3037] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:14.866371 17407 raft_consensus.cc:504] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.866994 17407 leader_election.cc:288] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [CANDIDATE]: Term 1 election: Requested vote from peers a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793), 170daff68d0a4494b6d2d7e9757ada3d (127.15.102.193:37159)
I20260620 18:44:14.867513 17336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34677da537b443d1961f643dd4c43bb0" candidate_uuid: "684f77b470a04fc198d2761ae73d51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6"
I20260620 18:44:14.867576 17058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34677da537b443d1961f643dd4c43bb0" candidate_uuid: "684f77b470a04fc198d2761ae73d51ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170daff68d0a4494b6d2d7e9757ada3d"
I20260620 18:44:14.867731 17336 raft_consensus.cc:3037] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:14.867813 17058 raft_consensus.cc:3037] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:14.869913 17058 raft_consensus.cc:2454] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 684f77b470a04fc198d2761ae73d51ec in term 1.
I20260620 18:44:14.870167 17336 raft_consensus.cc:2454] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 684f77b470a04fc198d2761ae73d51ec in term 1.
I20260620 18:44:14.870290 17125 leader_election.cc:302] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170daff68d0a4494b6d2d7e9757ada3d, 684f77b470a04fc198d2761ae73d51ec; no voters:
I20260620 18:44:14.870568 17407 raft_consensus.cc:2781] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:14.871024 17407 raft_consensus.cc:686] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 1 LEADER]: Becoming Leader. State: Replica: 684f77b470a04fc198d2761ae73d51ec, State: Running, Role: LEADER
I20260620 18:44:14.871584 17407 consensus_queue.cc:237] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } }
I20260620 18:44:14.876124 16894 catalog_manager.cc:5495] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 684f77b470a04fc198d2761ae73d51ec (127.15.102.194). New cstate: current_term: 1 leader_uuid: "684f77b470a04fc198d2761ae73d51ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } health_report { overall_health: UNKNOWN } } }
I20260620 18:44:14.924304 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:14.918623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.15.102.194:36673)
0620 18:44:14.918816 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:14.918822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:14.918899 (+ 77us) server_negotiation.cc:407] Connection header received
0620 18:44:14.919179 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:14.919184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:14.919335 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:14.919480 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:14.919808 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.920951 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.922278 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:14.922663 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:14.922786 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:14.923219 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:14.923814 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:14.923931 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:14.924043 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 18:44:14.924107 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:44:14.932564 17336 raft_consensus.cc:1261] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Refusing update from remote peer 684f77b470a04fc198d2761ae73d51ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:14.932557 17058 raft_consensus.cc:1261] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Refusing update from remote peer 684f77b470a04fc198d2761ae73d51ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:14.933511 17411 consensus_queue.cc:1035] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:14.933867 17407 consensus_queue.cc:1035] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:14.945544 17414 mvcc.cc:205] Tried to move back new op lower bound from 7298994400992223232 to 7298994400758005760. Current Snapshot: MvccSnapshot[applied={T|T < 7298994400992223232}]
I20260620 18:44:14.964836 16894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:14.965519 16894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:14.969349 16894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf: Sending DeleteTablet for 3 replicas of tablet 34677da537b443d1961f643dd4c43bb0
I20260620 18:44:14.970667 17172 tablet_service.cc:1514] Processing DeleteTablet for tablet 34677da537b443d1961f643dd4c43bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:14 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:45484
I20260620 18:44:14.970772 17316 tablet_service.cc:1514] Processing DeleteTablet for tablet 34677da537b443d1961f643dd4c43bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:14 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:59512
I20260620 18:44:14.970996 17033 tablet_service.cc:1514] Processing DeleteTablet for tablet 34677da537b443d1961f643dd4c43bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:14 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:51674
I20260620 18:44:14.973701 16894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:14.975159 16894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:14.981660 17425 tablet_replica.cc:330] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: stopping tablet replica
I20260620 18:44:14.982343 17425 raft_consensus.cc:2229] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:14.982950 17425 raft_consensus.cc:2258] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:14.985509 17424 tablet_replica.cc:330] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: stopping tablet replica
I20260620 18:44:14.986320 17425 ts_tablet_manager.cc:1905] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:14.986850 17424 raft_consensus.cc:2229] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:14.989568 17315 tablet_service.cc:1467] Processing CreateTablet for tablet 4394228e43bc4c47a4a18750c15feb04 (DEFAULT_TABLE table=test-table [id=12bf490eeb4141b18ea28a8dcd227a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:14.989916 17033 tablet_service.cc:1467] Processing CreateTablet for tablet 4394228e43bc4c47a4a18750c15feb04 (DEFAULT_TABLE table=test-table [id=12bf490eeb4141b18ea28a8dcd227a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:14.990165 17315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4394228e43bc4c47a4a18750c15feb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:14.990350 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4394228e43bc4c47a4a18750c15feb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:14.991577 17424 raft_consensus.cc:2258] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:14.993853 17424 ts_tablet_manager.cc:1905] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:14.996284 17171 tablet_service.cc:1467] Processing CreateTablet for tablet 4394228e43bc4c47a4a18750c15feb04 (DEFAULT_TABLE table=test-table [id=12bf490eeb4141b18ea28a8dcd227a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:14.996896 17171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4394228e43bc4c47a4a18750c15feb04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:14.997190 17402 tablet_bootstrap.cc:492] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: Bootstrap starting.
I20260620 18:44:14.998914 17402 tablet_bootstrap.cc:654] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:14.999045 17401 tablet_bootstrap.cc:492] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: Bootstrap starting.
I20260620 18:44:14.996929 17423 tablet_replica.cc:330] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: stopping tablet replica
I20260620 18:44:15.000856 17401 tablet_bootstrap.cc:654] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:15.001549 17425 ts_tablet_manager.cc:1918] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:15.001737 17425 log.cc:1196] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/34677da537b443d1961f643dd4c43bb0
I20260620 18:44:15.002354 17425 ts_tablet_manager.cc:1939] T 34677da537b443d1961f643dd4c43bb0 P 170daff68d0a4494b6d2d7e9757ada3d: Deleting consensus metadata
I20260620 18:44:15.002440 17423 raft_consensus.cc:2229] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:15.003163 17423 raft_consensus.cc:2258] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:15.005256 16882 catalog_manager.cc:4841] TS 170daff68d0a4494b6d2d7e9757ada3d (127.15.102.193:37159): tablet 34677da537b443d1961f643dd4c43bb0 (table test-table [id=7142a450288a4b40a332d76e0d14a7e6]) successfully deleted
I20260620 18:44:15.006348 17423 ts_tablet_manager.cc:1905] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:15.006567 17402 tablet_bootstrap.cc:492] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: No bootstrap required, opened a new log
I20260620 18:44:15.006708 17402 ts_tablet_manager.cc:1397] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 18:44:15.008842 17402 raft_consensus.cc:348] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.009138 17402 raft_consensus.cc:374] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:15.009285 17402 raft_consensus.cc:729] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170daff68d0a4494b6d2d7e9757ada3d, State: Initialized, Role: FOLLOWER
I20260620 18:44:15.009593 17402 consensus_queue.cc:260] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.010332 17402 ts_tablet_manager.cc:1428] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 18:44:15.012594 17400 tablet_bootstrap.cc:492] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: Bootstrap starting.
I20260620 18:44:15.014406 17400 tablet_bootstrap.cc:654] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:15.015687 17401 tablet_bootstrap.cc:492] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: No bootstrap required, opened a new log
I20260620 18:44:15.015849 17401 ts_tablet_manager.cc:1397] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20260620 18:44:15.016233 17423 ts_tablet_manager.cc:1918] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:15.016435 17423 log.cc:1196] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/34677da537b443d1961f643dd4c43bb0
I20260620 18:44:15.016978 17423 ts_tablet_manager.cc:1939] T 34677da537b443d1961f643dd4c43bb0 P a37b11cbe6cc4641a4d39ed15ca24eb6: Deleting consensus metadata
I20260620 18:44:15.016917 17401 raft_consensus.cc:348] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.017243 17401 raft_consensus.cc:374] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:15.017365 17401 raft_consensus.cc:729] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37b11cbe6cc4641a4d39ed15ca24eb6, State: Initialized, Role: FOLLOWER
I20260620 18:44:15.017699 17401 consensus_queue.cc:260] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.018625 17401 ts_tablet_manager.cc:1428] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:15.019078 16882 catalog_manager.cc:4841] TS a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793): tablet 34677da537b443d1961f643dd4c43bb0 (table test-table [id=7142a450288a4b40a332d76e0d14a7e6]) successfully deleted
I20260620 18:44:15.021363 17400 tablet_bootstrap.cc:492] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: No bootstrap required, opened a new log
I20260620 18:44:15.021610 17400 ts_tablet_manager.cc:1397] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 18:44:15.022645 17400 raft_consensus.cc:348] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.023098 17424 ts_tablet_manager.cc:1918] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:15.023386 17400 raft_consensus.cc:374] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:15.023540 17424 log.cc:1196] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/34677da537b443d1961f643dd4c43bb0
I20260620 18:44:15.023569 17400 raft_consensus.cc:729] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684f77b470a04fc198d2761ae73d51ec, State: Initialized, Role: FOLLOWER
I20260620 18:44:15.024003 17400 consensus_queue.cc:260] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.024214 17424 ts_tablet_manager.cc:1939] T 34677da537b443d1961f643dd4c43bb0 P 684f77b470a04fc198d2761ae73d51ec: Deleting consensus metadata
I20260620 18:44:15.024619 17400 ts_tablet_manager.cc:1428] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 18:44:15.026324 16881 catalog_manager.cc:4841] TS 684f77b470a04fc198d2761ae73d51ec (127.15.102.194:36673): tablet 34677da537b443d1961f643dd4c43bb0 (table test-table [id=7142a450288a4b40a332d76e0d14a7e6]) successfully deleted
I20260620 18:44:15.147486 17408 raft_consensus.cc:482] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:15.147761 17408 raft_consensus.cc:504] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.149415 17408 leader_election.cc:288] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 684f77b470a04fc198d2761ae73d51ec (127.15.102.194:36673), a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793)
I20260620 18:44:15.158169 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.149905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:50883 (local address 127.15.102.194:36673)
0620 18:44:15.150135 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.150144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.150914 (+ 770us) server_negotiation.cc:407] Connection header received
0620 18:44:15.151220 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.151227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.151333 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.151396 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:15.152271 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.153084 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.155692 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.156988 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.157136 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.157214 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.157699 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.157836 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:44:15.158357 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.150731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:48109 (local address 127.15.102.195:45793)
0620 18:44:15.150920 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.150927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.151123 (+ 196us) server_negotiation.cc:407] Connection header received
0620 18:44:15.151389 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.151395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.151465 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.151565 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:15.152371 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.153489 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.155554 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.156775 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.157219 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.157301 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.157983 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.158104 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:44:15.159000 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.150260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:45793 (local address 127.15.102.193:48109)
0620 18:44:15.151019 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:15.151033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:15.151185 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:15.151593 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:15.151597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:15.151633 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:15.152114 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.152123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.153614 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.153619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.155408 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.155419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.157632 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.157638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.157806 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.157835 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:44:15.158548 (+ 713us) client_negotiation.cc:240] Negotiation successful
0620 18:44:15.158683 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":17,"spinlock_wait_cycles":1792,"thread_start_us":863,"threads_started":1}
I20260620 18:44:15.159351 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.149632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:36673 (local address 127.15.102.193:50883)
0620 18:44:15.150794 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:15.150817 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:15.151052 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:15.151596 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:15.151601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:15.151643 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:15.152093 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.152107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.153227 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.153233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.155522 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.155537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.157253 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.157261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.157456 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.157492 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:44:15.158713 (+ 1221us) client_negotiation.cc:240] Negotiation successful
0620 18:44:15.158851 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":41,"spinlock_wait_cycles":7040,"thread_start_us":729,"threads_started":1}
I20260620 18:44:15.160177 17336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4394228e43bc4c47a4a18750c15feb04" candidate_uuid: "170daff68d0a4494b6d2d7e9757ada3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" is_pre_election: true
I20260620 18:44:15.160437 17336 raft_consensus.cc:2454] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170daff68d0a4494b6d2d7e9757ada3d in term 0.
I20260620 18:44:15.160940 17192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4394228e43bc4c47a4a18750c15feb04" candidate_uuid: "170daff68d0a4494b6d2d7e9757ada3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684f77b470a04fc198d2761ae73d51ec" is_pre_election: true
I20260620 18:44:15.161075 16985 leader_election.cc:302] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170daff68d0a4494b6d2d7e9757ada3d, a37b11cbe6cc4641a4d39ed15ca24eb6; no voters:
I20260620 18:44:15.161404 17192 raft_consensus.cc:2454] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170daff68d0a4494b6d2d7e9757ada3d in term 0.
I20260620 18:44:15.161510 17408 raft_consensus.cc:2781] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:15.161628 17408 raft_consensus.cc:482] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:15.161720 17408 raft_consensus.cc:3037] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:15.164847 17408 raft_consensus.cc:504] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.165518 17408 leader_election.cc:288] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [CANDIDATE]: Term 1 election: Requested vote from peers 684f77b470a04fc198d2761ae73d51ec (127.15.102.194:36673), a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793)
I20260620 18:44:15.165997 17336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4394228e43bc4c47a4a18750c15feb04" candidate_uuid: "170daff68d0a4494b6d2d7e9757ada3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6"
I20260620 18:44:15.166204 17336 raft_consensus.cc:3037] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:15.167448 17192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4394228e43bc4c47a4a18750c15feb04" candidate_uuid: "170daff68d0a4494b6d2d7e9757ada3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684f77b470a04fc198d2761ae73d51ec"
I20260620 18:44:15.167660 17192 raft_consensus.cc:3037] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:15.169052 17336 raft_consensus.cc:2454] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170daff68d0a4494b6d2d7e9757ada3d in term 1.
I20260620 18:44:15.170032 17192 raft_consensus.cc:2454] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170daff68d0a4494b6d2d7e9757ada3d in term 1.
I20260620 18:44:15.170403 16985 leader_election.cc:302] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170daff68d0a4494b6d2d7e9757ada3d, a37b11cbe6cc4641a4d39ed15ca24eb6; no voters:
I20260620 18:44:15.170684 17408 raft_consensus.cc:2781] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:15.171128 17408 raft_consensus.cc:686] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 LEADER]: Becoming Leader. State: Replica: 170daff68d0a4494b6d2d7e9757ada3d, State: Running, Role: LEADER
I20260620 18:44:15.171546 17408 consensus_queue.cc:237] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.175689 16894 catalog_manager.cc:5495] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 170daff68d0a4494b6d2d7e9757ada3d (127.15.102.193). New cstate: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:44:15 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981052, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:15.245949 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.230064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.15.102.254:40005)
0620 18:44:15.231586 (+ 1522us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.231593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.231618 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:44:15.231659 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.231663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.231715 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.231841 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:15.232935 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.233802 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.235296 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.235827 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.235983 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.240214 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:15.240223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:15.240228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:15.240278 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:15.243211 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.243939 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.243944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.243947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.244065 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.244429 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.244434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.244437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.244713 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:15.244869 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.245252 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.245569 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":1378,"thread_start_us":1212,"threads_started":1}
I20260620 18:44:15.245947 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.230026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40005 (local address 127.0.0.1:38626)
0620 18:44:15.231220 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:15.231276 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:15.231463 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:15.231864 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:15.231869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:15.232560 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:15.232761 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.232771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.233960 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.233966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.235125 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.235137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.236001 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.236008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.236254 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.236986 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:15.237004 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:15.239995 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:15.243398 (+ 3403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:15.243410 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:15.243419 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:15.243813 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:15.244170 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:15.244178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:15.244181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:15.244317 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:15.244844 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:15.244851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:15.245012 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 18:44:15.245168 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 18:44:15.245462 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":52,"thread_start_us":849,"threads_started":1}
I20260620 18:44:15.252400 16894 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:38626:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:44:15.253492 16894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:15.260601 17033 tablet_service.cc:1467] Processing CreateTablet for tablet 69922a7885b34e85aebc548ba2df4380 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d64eee8cea7460b89146f7ddaca0003]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:44:15.261219 17033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69922a7885b34e85aebc548ba2df4380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:15.261236 17171 tablet_service.cc:1467] Processing CreateTablet for tablet 69922a7885b34e85aebc548ba2df4380 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d64eee8cea7460b89146f7ddaca0003]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:44:15.261197 17316 tablet_service.cc:1467] Processing CreateTablet for tablet 69922a7885b34e85aebc548ba2df4380 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4d64eee8cea7460b89146f7ddaca0003]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:44:15.261770 17171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69922a7885b34e85aebc548ba2df4380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:15.261761 17316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69922a7885b34e85aebc548ba2df4380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:15.269166 17402 tablet_bootstrap.cc:492] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: Bootstrap starting.
I20260620 18:44:15.270716 17400 tablet_bootstrap.cc:492] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: Bootstrap starting.
I20260620 18:44:15.270889 17402 tablet_bootstrap.cc:654] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:15.271498 17401 tablet_bootstrap.cc:492] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: Bootstrap starting.
I20260620 18:44:15.272678 17400 tablet_bootstrap.cc:654] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:15.273839 17401 tablet_bootstrap.cc:654] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:15.284003 17401 tablet_bootstrap.cc:492] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: No bootstrap required, opened a new log
I20260620 18:44:15.284394 17401 ts_tablet_manager.cc:1397] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260620 18:44:15.285733 17401 raft_consensus.cc:348] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.285960 17402 tablet_bootstrap.cc:492] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: No bootstrap required, opened a new log
I20260620 18:44:15.286128 17402 ts_tablet_manager.cc:1397] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.015s
I20260620 18:44:15.286134 17401 raft_consensus.cc:374] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:15.286659 17401 raft_consensus.cc:729] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a37b11cbe6cc4641a4d39ed15ca24eb6, State: Initialized, Role: FOLLOWER
I20260620 18:44:15.287004 17401 consensus_queue.cc:260] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.287770 17401 ts_tablet_manager.cc:1428] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:44:15.287824 17400 tablet_bootstrap.cc:492] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: No bootstrap required, opened a new log
I20260620 18:44:15.287602 17406 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.288033 17400 ts_tablet_manager.cc:1397] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s
I20260620 18:44:15.288064 17406 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:15.288727 17402 raft_consensus.cc:348] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.289054 17402 raft_consensus.cc:374] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:15.289172 17402 raft_consensus.cc:729] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170daff68d0a4494b6d2d7e9757ada3d, State: Initialized, Role: FOLLOWER
I20260620 18:44:15.289315 17400 raft_consensus.cc:348] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.289588 17400 raft_consensus.cc:374] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:15.289713 17400 raft_consensus.cc:729] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684f77b470a04fc198d2761ae73d51ec, State: Initialized, Role: FOLLOWER
I20260620 18:44:15.289490 17402 consensus_queue.cc:260] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.289981 17400 consensus_queue.cc:260] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.290179 17408 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.290462 17408 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:15.290673 17400 ts_tablet_manager.cc:1428] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 18:44:15.291128 17411 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.291532 17411 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:15.293145 17402 ts_tablet_manager.cc:1428] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260620 18:44:15.339828 17408 raft_consensus.cc:482] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:15.340116 17408 raft_consensus.cc:504] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.340777 17408 leader_election.cc:288] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 684f77b470a04fc198d2761ae73d51ec (127.15.102.194:36673), a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793)
I20260620 18:44:15.341707 17192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69922a7885b34e85aebc548ba2df4380" candidate_uuid: "170daff68d0a4494b6d2d7e9757ada3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684f77b470a04fc198d2761ae73d51ec" is_pre_election: true
I20260620 18:44:15.341835 17336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69922a7885b34e85aebc548ba2df4380" candidate_uuid: "170daff68d0a4494b6d2d7e9757ada3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" is_pre_election: true
I20260620 18:44:15.342041 17192 raft_consensus.cc:2454] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170daff68d0a4494b6d2d7e9757ada3d in term 0.
I20260620 18:44:15.342144 17336 raft_consensus.cc:2454] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170daff68d0a4494b6d2d7e9757ada3d in term 0.
I20260620 18:44:15.342466 16984 leader_election.cc:302] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170daff68d0a4494b6d2d7e9757ada3d, 684f77b470a04fc198d2761ae73d51ec; no voters:
I20260620 18:44:15.342775 17408 raft_consensus.cc:2781] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:15.342907 17408 raft_consensus.cc:482] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:15.343050 17408 raft_consensus.cc:3037] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:15.345012 17408 raft_consensus.cc:504] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.345616 17408 leader_election.cc:288] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [CANDIDATE]: Term 1 election: Requested vote from peers 684f77b470a04fc198d2761ae73d51ec (127.15.102.194:36673), a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793)
I20260620 18:44:15.346076 17192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69922a7885b34e85aebc548ba2df4380" candidate_uuid: "170daff68d0a4494b6d2d7e9757ada3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684f77b470a04fc198d2761ae73d51ec"
I20260620 18:44:15.346129 17336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69922a7885b34e85aebc548ba2df4380" candidate_uuid: "170daff68d0a4494b6d2d7e9757ada3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6"
I20260620 18:44:15.346336 17336 raft_consensus.cc:3037] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:15.346365 17192 raft_consensus.cc:3037] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:15.349305 17336 raft_consensus.cc:2454] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170daff68d0a4494b6d2d7e9757ada3d in term 1.
I20260620 18:44:15.349412 17192 raft_consensus.cc:2454] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170daff68d0a4494b6d2d7e9757ada3d in term 1.
I20260620 18:44:15.349786 16985 leader_election.cc:302] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170daff68d0a4494b6d2d7e9757ada3d, a37b11cbe6cc4641a4d39ed15ca24eb6; no voters:
I20260620 18:44:15.350049 17408 raft_consensus.cc:2781] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:15.350258 17408 raft_consensus.cc:686] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 LEADER]: Becoming Leader. State: Replica: 170daff68d0a4494b6d2d7e9757ada3d, State: Running, Role: LEADER
I20260620 18:44:15.350521 17408 consensus_queue.cc:237] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } }
I20260620 18:44:15.352317 17431 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: TxnStatusTablet state changed. Reason: New leader 170daff68d0a4494b6d2d7e9757ada3d. Latest consensus state: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.352689 17431 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:44:15.353303 17444 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:44:15.353585 17444 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:44:15.354859 16896 catalog_manager.cc:5495] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 170daff68d0a4494b6d2d7e9757ada3d (127.15.102.193). New cstate: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } health_report { overall_health: UNKNOWN } } }
I20260620 18:44:15.388069 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.381245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.15.102.193:37159)
0620 18:44:15.382424 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.382428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.382442 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:44:15.382515 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.382518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.382584 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.382749 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:15.383132 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.384045 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.385524 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.385999 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.386148 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.386613 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:15.387360 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:15.387532 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.387642 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.387719 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":950,"threads_started":1}
I20260620 18:44:15.388053 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.381252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37159 (local address 127.0.0.1:51688)
0620 18:44:15.381674 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:15.381700 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:15.381887 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:15.382768 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:15.382773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:15.382785 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:44:15.382962 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.382971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.384172 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.384180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.385335 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.385346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.386161 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.386167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.386341 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.386394 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:44:15.387476 (+ 1082us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:44:15.387502 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 18:44:15.387620 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:44:15.387762 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260620 18:44:15.392225 17336 raft_consensus.cc:1261] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Refusing update from remote peer 170daff68d0a4494b6d2d7e9757ada3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:15.392670 17192 raft_consensus.cc:1261] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Refusing update from remote peer 170daff68d0a4494b6d2d7e9757ada3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:15.393232 17408 consensus_queue.cc:1035] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:15.393769 17408 consensus_queue.cc:1035] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:15.409758 17411 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: TxnStatusTablet state changed. Reason: New leader 170daff68d0a4494b6d2d7e9757ada3d. Latest consensus state: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.410209 17411 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:15.413123 17406 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: TxnStatusTablet state changed. Reason: New leader 170daff68d0a4494b6d2d7e9757ada3d. Latest consensus state: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.413378 17406 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:15.414878 17431 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.415129 17431 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:44:15.415711 17431 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.415941 17431 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P 170daff68d0a4494b6d2d7e9757ada3d: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:44:15.416217 17416 mvcc.cc:205] Tried to move back new op lower bound from 7298994402879066112 to 7298994402718158848. Current Snapshot: MvccSnapshot[applied={T|T < 7298994402879066112 or (T in {7298994402879066112})}]
I20260620 18:44:15.419019 17411 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.419708 17411 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P 684f77b470a04fc198d2761ae73d51ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:15.422032 17406 tablet_replica.cc:439] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "170daff68d0a4494b6d2d7e9757ada3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "170daff68d0a4494b6d2d7e9757ada3d" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37159 } } peers { permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 } } peers { permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 } } }
I20260620 18:44:15.422273 17406 tablet_replica.cc:442] T 69922a7885b34e85aebc548ba2df4380 P a37b11cbe6cc4641a4d39ed15ca24eb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:15.441879 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.435434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.15.102.193:37159)
0620 18:44:15.435717 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.435726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.436342 (+ 616us) server_negotiation.cc:407] Connection header received
0620 18:44:15.436564 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.436571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.436640 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.436753 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:15.437242 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.438386 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.439938 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.440499 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.440645 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.441037 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:15.441133 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:15.441233 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.441454 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.441536 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260620 18:44:15.478552 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.455311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:49273 (local address 127.15.102.193:37159)
0620 18:44:15.459513 (+ 4202us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.459520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.460568 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 18:44:15.471605 (+ 11037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.471611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.471699 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.471832 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:15.472435 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.473706 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.475990 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.476738 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.476867 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.477629 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:15.477783 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:15.477855 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.478070 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.478162 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 18:44:15.479468 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.455331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37159 (local address 127.15.102.193:49273)
0620 18:44:15.460478 (+ 5147us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:15.460498 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:15.471403 (+ 10905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:15.471844 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:15.471849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:15.471863 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:44:15.472222 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.472234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.473853 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.473858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.475826 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:15.475844 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.477097 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.477104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:15.477323 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.477431 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:44:15.477930 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:44:15.477945 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:44:15.478843 (+ 898us) client_negotiation.cc:240] Negotiation successful
0620 18:44:15.478988 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4697,"thread_start_us":4365,"threads_started":1}
I20260620 18:44:15.490370 17192 raft_consensus.cc:1261] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Refusing update from remote peer 170daff68d0a4494b6d2d7e9757ada3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:15.490816 17336 raft_consensus.cc:1261] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Refusing update from remote peer 170daff68d0a4494b6d2d7e9757ada3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:15.490957 17408 consensus_queue.cc:1035] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "684f77b470a04fc198d2761ae73d51ec" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:15.491349 17431 consensus_queue.cc:1035] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a37b11cbe6cc4641a4d39ed15ca24eb6" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:15.505084 17415 mvcc.cc:205] Tried to move back new op lower bound from 7298994403283722240 to 7298994401988542464. Current Snapshot: MvccSnapshot[applied={T|T < 7298994403283722240}]
I20260620 18:44:15.556361 16895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:15.556663 16895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:15.560181 16895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7e1665024374d17bb64c8924522d5cf: Sending DeleteTablet for 3 replicas of tablet 4394228e43bc4c47a4a18750c15feb04
I20260620 18:44:15.561187 17171 tablet_service.cc:1514] Processing DeleteTablet for tablet 4394228e43bc4c47a4a18750c15feb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:15 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:45484
I20260620 18:44:15.561245 17316 tablet_service.cc:1514] Processing DeleteTablet for tablet 4394228e43bc4c47a4a18750c15feb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:15 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:59512
I20260620 18:44:15.561983 17032 tablet_service.cc:1514] Processing DeleteTablet for tablet 4394228e43bc4c47a4a18750c15feb04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:15 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:51674
I20260620 18:44:15.576007 17461 tablet_replica.cc:330] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: stopping tablet replica
I20260620 18:44:15.576247 17461 raft_consensus.cc:2229] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:15.576952 17461 raft_consensus.cc:2258] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:15.578207 17461 ts_tablet_manager.cc:1905] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:15.578707 17460 tablet_replica.cc:330] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: stopping tablet replica
I20260620 18:44:15.579423 17460 raft_consensus.cc:2229] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:15.591547 17460 raft_consensus.cc:2258] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:15.593335 17459 tablet_replica.cc:330] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: stopping tablet replica
I20260620 18:44:15.593559 17459 raft_consensus.cc:2229] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:15.593786 17459 raft_consensus.cc:2258] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:15.594988 17460 ts_tablet_manager.cc:1905] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:44:15 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:15.602922 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.577184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.15.102.193:37159)
0620 18:44:15.577458 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.577472 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.581311 (+ 3839us) server_negotiation.cc:407] Connection header received
0620 18:44:15.581902 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.581909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.582018 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.582130 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:15.584832 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.586235 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.588872 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.589913 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.590037 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.596146 (+ 6109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:15.596159 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:15.596176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:15.596210 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:15.599653 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.600534 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.600548 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.600550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.600688 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.601097 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.601102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.601107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.601487 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:15.601860 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.602215 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.602466 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 18:44:15.603799 17098 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51692
I20260620 18:44:15.608266 17461 ts_tablet_manager.cc:1918] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:15.608620 17461 log.cc:1196] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/4394228e43bc4c47a4a18750c15feb04
I20260620 18:44:15.609277 17461 ts_tablet_manager.cc:1939] T 4394228e43bc4c47a4a18750c15feb04 P 170daff68d0a4494b6d2d7e9757ada3d: Deleting consensus metadata
I20260620 18:44:15.610900 16882 catalog_manager.cc:4841] TS 170daff68d0a4494b6d2d7e9757ada3d (127.15.102.193:37159): tablet 4394228e43bc4c47a4a18750c15feb04 (table test-table [id=12bf490eeb4141b18ea28a8dcd227a3b]) successfully deleted
I20260620 18:44:15.611996 17459 ts_tablet_manager.cc:1905] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:15.618961 17459 ts_tablet_manager.cc:1918] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:15.619180 17459 log.cc:1196] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/4394228e43bc4c47a4a18750c15feb04
I20260620 18:44:15.620136 17459 ts_tablet_manager.cc:1939] T 4394228e43bc4c47a4a18750c15feb04 P 684f77b470a04fc198d2761ae73d51ec: Deleting consensus metadata
I20260620 18:44:15.621268 16881 catalog_manager.cc:4841] TS 684f77b470a04fc198d2761ae73d51ec (127.15.102.194:36673): tablet 4394228e43bc4c47a4a18750c15feb04 (table test-table [id=12bf490eeb4141b18ea28a8dcd227a3b]) successfully deleted
I20260620 18:44:15.624202 17460 ts_tablet_manager.cc:1918] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:15.624580 17460 log.cc:1196] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/4394228e43bc4c47a4a18750c15feb04
I20260620 18:44:15.625123 17460 ts_tablet_manager.cc:1939] T 4394228e43bc4c47a4a18750c15feb04 P a37b11cbe6cc4641a4d39ed15ca24eb6: Deleting consensus metadata
I20260620 18:44:15.626194 16882 catalog_manager.cc:4841] TS a37b11cbe6cc4641a4d39ed15ca24eb6 (127.15.102.195:45793): tablet 4394228e43bc4c47a4a18750c15feb04 (table test-table [id=12bf490eeb4141b18ea28a8dcd227a3b]) successfully deleted
I20260620 18:44:15.629979 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.607386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.15.102.254:40005)
0620 18:44:15.607612 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.607619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.614425 (+ 6806us) server_negotiation.cc:407] Connection header received
0620 18:44:15.614621 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.614627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.614709 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.614810 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:15.616585 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.617716 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.619626 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.620257 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.620350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.623449 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:15.623456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:15.623461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:15.623500 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:15.627122 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.627910 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.627915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.627919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.628056 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.628475 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.628480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.628483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.628832 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:15.629075 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.629437 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.629658 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260620 18:44:15.630517 16895 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38640
I20260620 18:44:15.632683 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16967
I20260620 18:44:15.650060 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.635323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.15.102.193:37159)
0620 18:44:15.635512 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.635518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.635536 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:44:15.635594 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.635599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.635667 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.635766 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:15.637220 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.638395 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.640214 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.640943 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.641040 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.643831 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:15.643835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:15.643839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:15.643873 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:15.646814 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.647609 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.647614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.647618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.647797 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.648220 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.648225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.648228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.648505 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:15.648795 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.649079 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.649289 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:44:15.650617 17098 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51706
W20260620 18:44:15.651172 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16967: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:15.651415 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 16967
I20260620 18:44:15.671473 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17107
Jun 20 18:44:15 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:44:15.685760 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.672137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.15.102.194:36673)
0620 18:44:15.672596 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.672600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.672615 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:44:15.672664 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.672667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.672724 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.672847 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:15.674012 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.674718 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.676048 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.676471 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.676608 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.680491 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:15.680504 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:15.680519 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:15.680554 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:15.683080 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.683749 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.683756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.683758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.683877 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.684271 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.684280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.684282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.684601 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:15.684909 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.685174 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.685425 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":242,"threads_started":1}
W20260620 18:44:15.686404 17237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45508
W20260620 18:44:15.687058 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17107: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:15.687198 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 17107
I20260620 18:44:15.708619 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17246
Jun 20 18:44:15 dist-test-slave-zfp8 krb5kdc[16851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981054, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:44:15.723162 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:15.709335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.15.102.195:45793)
0620 18:44:15.709810 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:15.709814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:15.709829 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:44:15.709881 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:15.709884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:15.709947 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:15.710068 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:15.711312 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.712013 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.713277 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:15.713697 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:15.713837 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:15.718191 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:15.718205 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:15.718221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:15.718255 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:15.720932 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.721537 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.721544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.721546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.721673 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:15.721971 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:15.721974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:15.721976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:15.722314 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:15.722516 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:15.722767 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 18:44:15.722924 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":214,"threads_started":1}
W20260620 18:44:15.723901 17376 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59524
W20260620 18:44:15.724519 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17246: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:15.724648 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 17246
I20260620 18:44:15.741761 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 16864
I20260620 18:44:15.742944 16936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:15.856980 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 16864
2026-06-20T18:44:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4659 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:15 dist-test-slave-zfp8 krb5kdc[17472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:15 dist-test-slave-zfp8 krb5kdc[17472](info): set up 2 sockets
Jun 20 18:44:15 dist-test-slave-zfp8 krb5kdc[17472](info): commencing operation
krb5kdc: starting...
W20260620 18:44:16.947654 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:17 dist-test-slave-zfp8 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:17.129166 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46117
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:46117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:17.452174 17485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:17.452488 17485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:17.452649 17485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:17.459976 17485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:17.460083 17485 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:44:17.460177 17485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:17.460289 17485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:17.460393 17485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:17.460506 17485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:17.470440 17485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:46117
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:17.471803 17485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:17.473125 17485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:17.484951 17490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:17.485098 17491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:17.485044 17493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:17.486176 17485 server_base.cc:1034] running on GCE node
I20260620 18:44:17.486991 17485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:17.488512 17485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:17.489741 17485 hybrid_clock.cc:648] HybridClock initialized: now 1781981057489679 us; error 31 us; skew 500 ppm
I20260620 18:44:17.490120 17485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:17 dist-test-slave-zfp8 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981057, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:17.493225 17485 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:17.494594 17485 webserver.cc:460] Webserver started at http://127.15.102.254:46177/ using document root <none> and password file <none>
I20260620 18:44:17.495325 17485 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:17.495419 17485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:17.495672 17485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:17.498497 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d"
format_stamp: "Formatted at 2026-06-20 18:44:17 on dist-test-slave-zfp8"
server_key: "70dbe990b1e0e53611a2703406798ea2"
server_key_iv: "0cefecee26d5fc8f0b3182c0ab444ca3"
server_key_version: "encryptionkey@0"
I20260620 18:44:17.499400 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d"
format_stamp: "Formatted at 2026-06-20 18:44:17 on dist-test-slave-zfp8"
server_key: "70dbe990b1e0e53611a2703406798ea2"
server_key_iv: "0cefecee26d5fc8f0b3182c0ab444ca3"
server_key_version: "encryptionkey@0"
I20260620 18:44:17.504611 17485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:44:17.508011 17500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:17.509461 17485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:17.509622 17485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d"
format_stamp: "Formatted at 2026-06-20 18:44:17 on dist-test-slave-zfp8"
server_key: "70dbe990b1e0e53611a2703406798ea2"
server_key_iv: "0cefecee26d5fc8f0b3182c0ab444ca3"
server_key_version: "encryptionkey@0"
I20260620 18:44:17.509933 17485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:17.528909 17485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:17.533210 17485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:17.533435 17485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:17.555269 17485 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:46117
I20260620 18:44:17.555368 17561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:46117 every 8 connection(s)
I20260620 18:44:17.556675 17485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:17.558254 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 17485
I20260620 18:44:17.558466 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:17.559140 15771 external_mini_cluster.cc:1428] Setting key 5af1c3ba9bcacf1c3b885a1e2c53a488
I20260620 18:44:17.560956 17562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:44:17 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:17.574772 17562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: Bootstrap starting.
I20260620 18:44:17.579074 17562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:17.580394 17562 log.cc:824] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:17.583730 17562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: No bootstrap required, opened a new log
I20260620 18:44:17.583882 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:17.561137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.15.102.254:46117)
0620 18:44:17.561874 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:17.561891 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:17.561944 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:44:17.564032 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:17.564059 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:17.564607 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:17.565078 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:17.566425 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:17.568526 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:17.569765 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:17.570793 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:17.570996 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:17.575351 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:17.575361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:17.575382 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:17.575438 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:17.579458 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:17.580184 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:17.580195 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:17.580206 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:17.580421 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:17.580799 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:17.580807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:17.580809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:17.581438 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:17.581688 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:17.582148 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 18:44:17.582573 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":313,"threads_started":1}
I20260620 18:44:17.591761 17562 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46117 } }
I20260620 18:44:17.592104 17562 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:17.592178 17562 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f4a036e6fe430fa4b25d34bf4faf5d, State: Initialized, Role: FOLLOWER
I20260620 18:44:17.592653 17562 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46117 } }
I20260620 18:44:17.592837 17562 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:17.592937 17562 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:17.593066 17562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:17.595073 17562 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46117 } }
I20260620 18:44:17.595600 17562 leader_election.cc:302] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9f4a036e6fe430fa4b25d34bf4faf5d; no voters:
I20260620 18:44:17.596206 17562 leader_election.cc:288] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:17.596374 17567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:17.597601 17567 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [term 1 LEADER]: Becoming Leader. State: Replica: e9f4a036e6fe430fa4b25d34bf4faf5d, State: Running, Role: LEADER
I20260620 18:44:17.598129 17567 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46117 } }
I20260620 18:44:17.598433 17562 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:17.601212 17568 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46117 } } }
I20260620 18:44:17.601688 17568 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:17.601228 17569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f4a036e6fe430fa4b25d34bf4faf5d. Latest consensus state: current_term: 1 leader_uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9f4a036e6fe430fa4b25d34bf4faf5d" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46117 } } }
I20260620 18:44:17.602183 17569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:17.604204 17576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:17.609535 17576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:17.618189 17576 catalog_manager.cc:1347] Generated new cluster ID: b68bec4ffa964a72870124a333cdbea4
I20260620 18:44:17.618327 17576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:17.631009 17576 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:17.632354 17576 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:17.651036 17576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: Generated new TSK 0
I20260620 18:44:17.652069 17576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:17.708851 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46117
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:18.032605 17588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:18.032936 17588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:18.033120 17588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:18.040465 17588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:18.040614 17588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:18.040772 17588 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:44:18.040875 17588 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:44:18.041015 17588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:18.050853 17588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:18.052193 17588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:18.053566 17588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:18.066841 17594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:18.066871 17596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:18.066996 17593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:18.067277 17588 server_base.cc:1034] running on GCE node
I20260620 18:44:18.068069 17588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:18.068946 17588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:18.070220 17588 hybrid_clock.cc:648] HybridClock initialized: now 1781981058070102 us; error 57 us; skew 500 ppm
I20260620 18:44:18.070552 17588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:18 dist-test-slave-zfp8 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981058, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:18.073813 17588 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:18.075487 17588 webserver.cc:460] Webserver started at http://127.15.102.193:45355/ using document root <none> and password file <none>
I20260620 18:44:18.076164 17588 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:18.076283 17588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:18.076565 17588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:18.079654 17588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "f5eb54b34c054558ac49176d4a270275"
format_stamp: "Formatted at 2026-06-20 18:44:18 on dist-test-slave-zfp8"
server_key: "f0669d249516970d660344f26b0a22ba"
server_key_iv: "18ab609e656d95be4ebc0f0bc6aaa4fe"
server_key_version: "encryptionkey@0"
I20260620 18:44:18.080406 17588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "f5eb54b34c054558ac49176d4a270275"
format_stamp: "Formatted at 2026-06-20 18:44:18 on dist-test-slave-zfp8"
server_key: "f0669d249516970d660344f26b0a22ba"
server_key_iv: "18ab609e656d95be4ebc0f0bc6aaa4fe"
server_key_version: "encryptionkey@0"
I20260620 18:44:18.085975 17588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:18.089624 17603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:18.091025 17588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:18.091166 17588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "f5eb54b34c054558ac49176d4a270275"
format_stamp: "Formatted at 2026-06-20 18:44:18 on dist-test-slave-zfp8"
server_key: "f0669d249516970d660344f26b0a22ba"
server_key_iv: "18ab609e656d95be4ebc0f0bc6aaa4fe"
server_key_version: "encryptionkey@0"
I20260620 18:44:18.091426 17588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:18.106288 17588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:18.110278 17588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:18.110587 17588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:18.113289 17588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:18.113389 17588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:18.113583 17588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:18.113687 17588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:44:18 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981058, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:18.140391 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:18.114310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:44309 (local address 127.15.102.254:46117)
0620 18:44:18.114849 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:18.114858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:18.116109 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 18:44:18.118703 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:18.118713 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:18.118825 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:18.118979 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:18.121464 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.122397 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.125751 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.126181 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.126306 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:18.132379 (+ 6073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:18.132389 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:18.132397 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:18.132448 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:18.136033 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:18.137249 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:18.137253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:18.137255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:18.137400 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:18.138081 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:18.138085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:18.138088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:18.138440 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:18.138587 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:18.139909 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0620 18:44:18.140088 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":321,"threads_started":1}
I20260620 18:44:18.142956 17613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:18.115031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46117 (local address 127.15.102.193:44309)
0620 18:44:18.115936 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:18.115979 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:18.118268 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:18.119349 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:18.119365 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:18.120226 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:18.121222 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:18.121264 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.122597 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.122607 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:18.125559 (+ 2952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:18.125578 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.126459 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.126469 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:18.126831 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:18.127939 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:18.127978 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:18.131980 (+ 4002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:18.136207 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:18.136221 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:18.136248 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:18.137081 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:18.137524 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:18.137532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:18.137538 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:18.137851 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:18.138599 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:18.138611 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:18.139303 (+ 692us) client_negotiation.cc:769] Sending connection context
0620 18:44:18.139860 (+ 557us) client_negotiation.cc:240] Negotiation successful
0620 18:44:18.140646 (+ 786us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":383,"threads_started":1}
I20260620 18:44:18.165203 17588 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:35345
I20260620 18:44:18.165299 17723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:35345 every 8 connection(s)
I20260620 18:44:18.166623 17588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:18.167200 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:18.149819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:43339 (local address 127.15.102.254:46117)
0620 18:44:18.150001 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:18.150006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:18.150945 (+ 939us) server_negotiation.cc:407] Connection header received
0620 18:44:18.151218 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:18.151272 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:18.151357 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:18.151466 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:18.152917 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.154246 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.156501 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.156910 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.157052 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:18.160025 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:18.160035 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:18.160044 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:18.160098 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:18.164016 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:18.165008 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:18.165015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:18.165021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:18.165196 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:18.165666 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:18.165672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:18.165675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:18.166035 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:18.166199 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:18.166614 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 18:44:18.166834 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:44:18.167436 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:18.149918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46117 (local address 127.15.102.193:43339)
0620 18:44:18.150831 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:18.150854 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:18.151063 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:18.151495 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:18.151500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:18.152215 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:18.152707 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:18.152728 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.154445 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.154453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:18.156331 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:18.156347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.157111 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.157118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:18.157281 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:18.158036 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:18.158078 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:18.159730 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:18.164185 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:18.164193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:18.164203 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:18.164846 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:18.165363 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:18.165370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:18.165375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:18.165540 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:18.166161 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:18.166166 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:18.166360 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 18:44:18.166522 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 18:44:18.166887 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":740,"threads_started":1}
I20260620 18:44:18.169268 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 17588
I20260620 18:44:18.169519 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:18.170369 15771 external_mini_cluster.cc:1428] Setting key da4cb70ebf3cbd274c296ed841200890
I20260620 18:44:18.171940 17724 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46117
I20260620 18:44:18.172278 17724 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:18.173718 17724 heartbeater.cc:511] Master 127.15.102.254:46117 requested a full tablet report, sending...
I20260620 18:44:18.175841 17517 ts_manager.cc:194] Registered new tserver with Master: f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345)
I20260620 18:44:18.177330 17517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:44309
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:18.212388 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46117
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:18.541286 17728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:18.541566 17728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:18.541667 17728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:18.549048 17728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:18.549162 17728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:18.549247 17728 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:44:18.549299 17728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:44:18.549362 17728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:18.559094 17728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:18.560449 17728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:18.561856 17728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:18.575299 17736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:18.575343 17734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:18.575503 17733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:18.576418 17728 server_base.cc:1034] running on GCE node
I20260620 18:44:18.576993 17728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:18.577930 17728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:18.579126 17728 hybrid_clock.cc:648] HybridClock initialized: now 1781981058579062 us; error 53 us; skew 500 ppm
I20260620 18:44:18.579476 17728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:18 dist-test-slave-zfp8 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981058, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:18.582509 17728 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:18.584057 17728 webserver.cc:460] Webserver started at http://127.15.102.194:40723/ using document root <none> and password file <none>
I20260620 18:44:18.584731 17728 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:18.584882 17728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:18.585191 17728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:18.588109 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2"
format_stamp: "Formatted at 2026-06-20 18:44:18 on dist-test-slave-zfp8"
server_key: "09d31acc42f7344fad43a0c0752f376d"
server_key_iv: "9ae122972492deda1cd68605e15dad4b"
server_key_version: "encryptionkey@0"
I20260620 18:44:18.588881 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2"
format_stamp: "Formatted at 2026-06-20 18:44:18 on dist-test-slave-zfp8"
server_key: "09d31acc42f7344fad43a0c0752f376d"
server_key_iv: "9ae122972492deda1cd68605e15dad4b"
server_key_version: "encryptionkey@0"
I20260620 18:44:18.593868 17728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:18.597601 17743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:18.598949 17728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:18.599123 17728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2"
format_stamp: "Formatted at 2026-06-20 18:44:18 on dist-test-slave-zfp8"
server_key: "09d31acc42f7344fad43a0c0752f376d"
server_key_iv: "9ae122972492deda1cd68605e15dad4b"
server_key_version: "encryptionkey@0"
I20260620 18:44:18.599370 17728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:18.615767 17728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:18.619809 17728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:18.620101 17728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:18.622752 17728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:18.622884 17728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:18.623008 17728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:18.623134 17728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 18:44:18 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981058, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:18.646042 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:18.623107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:55697 (local address 127.15.102.254:46117)
0620 18:44:18.623306 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:18.623311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:18.624799 (+ 1488us) server_negotiation.cc:407] Connection header received
0620 18:44:18.626915 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:18.626921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:18.627029 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:18.627164 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:18.629324 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.630642 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.633184 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.633974 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.634117 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:18.639201 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:18.639206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:18.639209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:18.639287 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:18.641841 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:18.642949 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:18.642953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:18.642955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:18.643069 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:18.643632 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:18.643635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:18.643637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:18.643907 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:18.644029 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:18.645504 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0620 18:44:18.645727 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:44:18.647832 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:18.623818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46117 (local address 127.15.102.194:55697)
0620 18:44:18.624649 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:18.624690 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:18.626546 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:18.627464 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:18.627474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:18.628306 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:18.629108 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:18.629130 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.630799 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.630808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:18.633065 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:18.633080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.634258 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.634264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:18.634439 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:18.635339 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:18.635361 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:18.638944 (+ 3583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:18.642008 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:18.642019 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:18.642041 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:18.642809 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:18.643181 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:18.643189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:18.643196 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:18.643494 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:18.644038 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:18.644049 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:18.644599 (+ 550us) client_negotiation.cc:769] Sending connection context
0620 18:44:18.644958 (+ 359us) client_negotiation.cc:240] Negotiation successful
0620 18:44:18.645720 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":395,"threads_started":1}
I20260620 18:44:18.672132 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:18.657824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:52851 (local address 127.15.102.254:46117)
0620 18:44:18.658006 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:18.658013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:18.659186 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 18:44:18.659453 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:18.659461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:18.659546 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:18.659648 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:18.660978 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.661707 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.662993 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.663427 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.663553 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:18.666041 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:18.666045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:18.666048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:18.666077 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:18.669426 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:18.670121 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:18.670128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:18.670135 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:18.670288 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:18.670662 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:18.670669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:18.670675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:18.670983 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:18.671133 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:18.671594 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 18:44:18.671798 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:44:18.672267 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:18.657943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46117 (local address 127.15.102.194:52851)
0620 18:44:18.659057 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:18.659084 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:18.659275 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:18.659803 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:18.659806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:18.660514 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:18.660831 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:18.660840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.661845 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.661851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:18.662843 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:18.662853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:18.663641 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:18.663647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:18.663816 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:18.664601 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:18.664624 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:18.665818 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:18.669588 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:18.669591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:18.669594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:18.669985 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:18.670414 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:18.670423 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:18.670425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:18.670548 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:18.671099 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:18.671104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:18.671335 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 18:44:18.671499 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 18:44:18.671824 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":516,"threads_started":1}
I20260620 18:44:18.681133 17728 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:45285
I20260620 18:44:18.681185 17862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:45285 every 8 connection(s)
I20260620 18:44:18.682430 17728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:18.684397 17863 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46117
I20260620 18:44:18.684613 17863 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:18.685247 17863 heartbeater.cc:511] Master 127.15.102.254:46117 requested a full tablet report, sending...
I20260620 18:44:18.686501 17517 ts_manager.cc:194] Registered new tserver with Master: 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285)
I20260620 18:44:18.687325 17517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:55697
I20260620 18:44:18.690752 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 17728
I20260620 18:44:18.690887 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:18.691318 15771 external_mini_cluster.cc:1428] Setting key 23f930e668dd1e6587698aea5f051d47
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:18.728199 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46117
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:19.056712 17867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:19.057081 17867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:19.057279 17867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:19.065104 17867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:19.065279 17867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:19.065434 17867 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 18:44:19.065529 17867 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:44:19.065677 17867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:19.076077 17867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:46117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:19.077487 17867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:19.078938 17867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:19.093014 17872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:19.093029 17875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:19.093016 17873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:19.094012 17867 server_base.cc:1034] running on GCE node
I20260620 18:44:19.094679 17867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:19.095748 17867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:19.096987 17867 hybrid_clock.cc:648] HybridClock initialized: now 1781981059096923 us; error 63 us; skew 500 ppm
I20260620 18:44:19.097329 17867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:19 dist-test-slave-zfp8 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981059, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:19.100826 17867 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:19.102470 17867 webserver.cc:460] Webserver started at http://127.15.102.195:45731/ using document root <none> and password file <none>
I20260620 18:44:19.103315 17867 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:19.103515 17867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:19.103956 17867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:19.107002 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "809d6c5efa0349f684708986558bad8e"
format_stamp: "Formatted at 2026-06-20 18:44:19 on dist-test-slave-zfp8"
server_key: "295a90756880c7789c712628734f269c"
server_key_iv: "776dc57a18e46bb3939fb9361fb9113e"
server_key_version: "encryptionkey@0"
I20260620 18:44:19.107831 17867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "809d6c5efa0349f684708986558bad8e"
format_stamp: "Formatted at 2026-06-20 18:44:19 on dist-test-slave-zfp8"
server_key: "295a90756880c7789c712628734f269c"
server_key_iv: "776dc57a18e46bb3939fb9361fb9113e"
server_key_version: "encryptionkey@0"
I20260620 18:44:19.113299 17867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:44:19.116874 17882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:19.118198 17867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 18:44:19.118364 17867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "809d6c5efa0349f684708986558bad8e"
format_stamp: "Formatted at 2026-06-20 18:44:19 on dist-test-slave-zfp8"
server_key: "295a90756880c7789c712628734f269c"
server_key_iv: "776dc57a18e46bb3939fb9361fb9113e"
server_key_version: "encryptionkey@0"
I20260620 18:44:19.118558 17867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:19.134016 17867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:19.138005 17867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:19.138303 17867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:19.141059 17867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:19.141242 17867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:44:19.141398 17867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:19.141508 17867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:44:19 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981059, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:19.165146 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.141404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:41155 (local address 127.15.102.254:46117)
0620 18:44:19.141600 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.141607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.143040 (+ 1433us) server_negotiation.cc:407] Connection header received
0620 18:44:19.145172 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.145181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.145277 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.145439 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:19.147740 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.149166 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.151759 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.152471 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.152640 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.157070 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:19.157077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:19.157083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:19.157124 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:19.160923 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:19.161991 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:19.161999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:19.162003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:19.162183 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:19.162925 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:19.162932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:19.162938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:19.163402 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:19.163576 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.164517 (+ 941us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.164755 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:44:19.166260 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.141971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46117 (local address 127.15.102.195:41155)
0620 18:44:19.142850 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.142894 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.144887 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.145814 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.145860 (+ 46us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.146639 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:19.147489 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.147546 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.149354 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.149363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.151591 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.151611 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.152743 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.152749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.152941 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.153640 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:19.153661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:19.156771 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:19.161125 (+ 4354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:19.161132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:19.161144 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:19.161835 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:19.162344 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:19.162353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:19.162359 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:19.162736 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:19.163554 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:19.163560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:19.163960 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.164400 (+ 440us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.164991 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":351,"threads_started":1}
I20260620 18:44:19.180600 17724 heartbeater.cc:503] Master 127.15.102.254:46117 was elected leader, sending a full tablet report...
I20260620 18:44:19.192385 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.175122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:40769 (local address 127.15.102.254:46117)
0620 18:44:19.175359 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.175367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.176374 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 18:44:19.176570 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.176576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.176681 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.176828 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:19.178248 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.178996 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.181718 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.182623 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.182745 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.185977 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:19.185985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:19.185992 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:19.186036 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:19.189462 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:19.190382 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:19.190387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:19.190391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:19.190523 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:19.190900 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:19.190912 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:19.190914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:19.191213 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:19.191411 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.191930 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.192133 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:44:19.192975 17954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.175165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46117 (local address 127.15.102.195:40769)
0620 18:44:19.176219 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.176244 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.176423 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.176860 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.176868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.177625 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:19.178080 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.178097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.179140 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.179148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.181495 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.181515 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.183109 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.183119 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.183416 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.184262 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:19.184294 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:19.185751 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:19.189607 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:19.189617 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:19.189621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:19.190252 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:19.190648 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:19.190654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:19.190657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:19.190784 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:19.191366 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:19.191374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:19.191651 (+ 277us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.191846 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.192303 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"thread_start_us":538,"threads_started":1}
I20260620 18:44:19.202207 17867 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:33445
I20260620 18:44:19.202324 18001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:33445 every 8 connection(s)
I20260620 18:44:19.203696 17867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:19.205770 18002 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46117
I20260620 18:44:19.205997 18002 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:19.206634 18002 heartbeater.cc:511] Master 127.15.102.254:46117 requested a full tablet report, sending...
I20260620 18:44:19.206734 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 17867
I20260620 18:44:19.206940 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:19.207501 15771 external_mini_cluster.cc:1428] Setting key 0370ba5f42aaed52b65b0c0259650cb6
I20260620 18:44:19.208899 17517 ts_manager.cc:194] Registered new tserver with Master: 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445)
I20260620 18:44:19.210049 17517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:41155
I20260620 18:44:19.212580 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:44:19 dist-test-slave-zfp8 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:44:19 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:19.258005 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.244374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.15.102.254:46117)
0620 18:44:19.244617 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.244626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.244802 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:44:19.245028 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.245033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.245098 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.245185 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:19.246610 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.247405 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.248817 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.249226 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.249363 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.253054 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:19.253058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:19.253062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:19.253096 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:19.255604 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:19.256277 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:19.256281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:19.256284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:19.256411 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:19.256725 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:19.256728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:19.256730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:19.256998 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:19.257168 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.257442 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.257597 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 18:44:19.262110 17516 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:19.269055 17516 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:19.302788 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.282742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:45285 (local address 127.0.0.1:54766)
0620 18:44:19.291929 (+ 9187us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.291948 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.292067 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.293223 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.293236 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.293285 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:19.293720 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.293735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.296093 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.296102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.299138 (+ 3036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.299155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.301694 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.301702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.301917 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.301982 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.302122 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.302371 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9026,"thread_start_us":374,"threads_started":1}
I20260620 18:44:19.303917 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.283461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.15.102.194:45285)
0620 18:44:19.291578 (+ 8117us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.291591 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.292214 (+ 623us) server_negotiation.cc:407] Connection header received
0620 18:44:19.292337 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.292345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.292659 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.292904 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:19.293978 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.295883 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.299355 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.301202 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.301377 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.301576 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.303215 (+ 1639us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.303511 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7952,"thread_start_us":7817,"threads_started":1}
I20260620 18:44:19.308259 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.282323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:33445 (local address 127.0.0.1:38700)
0620 18:44:19.283413 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.283450 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.283627 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.300578 (+ 16951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.300583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.300608 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:19.301384 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.301396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.303782 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.303787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.305706 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.305720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.307572 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.307577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.307739 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.307771 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.307838 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.307969 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":9299,"threads_started":1}
I20260620 18:44:19.307881 17793 tablet_service.cc:1467] Processing CreateTablet for tablet 476255c253364b23956d8ba323c0a686 (DEFAULT_TABLE table=test-table [id=ab362e91583b4c078ad4384cfb046b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:19.309392 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.282949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.15.102.195:33445)
0620 18:44:19.299955 (+ 17006us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.299964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.299997 (+ 33us) server_negotiation.cc:407] Connection header received
0620 18:44:19.300089 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.300097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.300348 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.301605 (+ 1257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:19.301904 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.303628 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.305933 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.307352 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.308676 (+ 1324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.308838 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.308912 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.309080 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16598,"thread_start_us":16725,"threads_started":1}
I20260620 18:44:19.309478 17793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476255c253364b23956d8ba323c0a686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:19.312922 17932 tablet_service.cc:1467] Processing CreateTablet for tablet 476255c253364b23956d8ba323c0a686 (DEFAULT_TABLE table=test-table [id=ab362e91583b4c078ad4384cfb046b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:19.314400 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476255c253364b23956d8ba323c0a686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:19.320850 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.292009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:35345 (local address 127.0.0.1:33292)
0620 18:44:19.292513 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.292537 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.292678 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.311742 (+ 19064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.311747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.311777 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:19.312144 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.312156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.315609 (+ 3453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.315615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.317754 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.317770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.320028 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.320035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.320223 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.320257 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.320350 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.320499 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":254,"threads_started":1}
I20260620 18:44:19.322749 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.292329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.15.102.193:35345)
0620 18:44:19.311131 (+ 18802us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.311140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.311171 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:44:19.311291 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.311299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.311545 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.311755 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:19.312343 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.315426 (+ 3083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.317952 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.319559 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.319731 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.319890 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.321084 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.321248 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18627,"thread_start_us":19583,"threads_started":1}
I20260620 18:44:19.324481 18021 tablet_bootstrap.cc:492] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Bootstrap starting.
I20260620 18:44:19.326732 17654 tablet_service.cc:1467] Processing CreateTablet for tablet 476255c253364b23956d8ba323c0a686 (DEFAULT_TABLE table=test-table [id=ab362e91583b4c078ad4384cfb046b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:19.327391 18021 tablet_bootstrap.cc:654] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:19.328171 17654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476255c253364b23956d8ba323c0a686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:19.328649 18021 log.cc:824] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:19.345950 18022 tablet_bootstrap.cc:492] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Bootstrap starting.
I20260620 18:44:19.350135 18021 tablet_bootstrap.cc:492] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: No bootstrap required, opened a new log
I20260620 18:44:19.350333 18024 tablet_bootstrap.cc:492] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Bootstrap starting.
I20260620 18:44:19.350461 18021 ts_tablet_manager.cc:1397] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.004s
I20260620 18:44:19.351765 18022 tablet_bootstrap.cc:654] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:19.353106 18022 log.cc:824] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:19.355782 18024 tablet_bootstrap.cc:654] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:19.357177 18024 log.cc:824] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:19.364672 18021 raft_consensus.cc:348] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.365170 18021 raft_consensus.cc:374] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:19.365299 18021 raft_consensus.cc:729] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8cb03b7c104cc4bf844d3e9d9f33f2, State: Initialized, Role: FOLLOWER
I20260620 18:44:19.366057 18022 tablet_bootstrap.cc:492] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: No bootstrap required, opened a new log
I20260620 18:44:19.366024 18021 consensus_queue.cc:260] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.366364 18022 ts_tablet_manager.cc:1397] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20260620 18:44:19.366477 18024 tablet_bootstrap.cc:492] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: No bootstrap required, opened a new log
I20260620 18:44:19.366860 18024 ts_tablet_manager.cc:1397] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260620 18:44:19.378690 18022 raft_consensus.cc:348] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.379184 18022 raft_consensus.cc:374] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:19.379312 18022 raft_consensus.cc:729] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d6c5efa0349f684708986558bad8e, State: Initialized, Role: FOLLOWER
I20260620 18:44:19.380125 17863 heartbeater.cc:503] Master 127.15.102.254:46117 was elected leader, sending a full tablet report...
I20260620 18:44:19.380080 18022 consensus_queue.cc:260] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.380944 18021 ts_tablet_manager.cc:1428] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent starting tablet: real 0.030s user 0.023s sys 0.008s
I20260620 18:44:19.384891 18024 raft_consensus.cc:348] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.385488 18024 raft_consensus.cc:374] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:19.385644 18024 raft_consensus.cc:729] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eb54b34c054558ac49176d4a270275, State: Initialized, Role: FOLLOWER
I20260620 18:44:19.386409 18024 consensus_queue.cc:260] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.388386 18022 ts_tablet_manager.cc:1428] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Time spent starting tablet: real 0.022s user 0.013s sys 0.009s
I20260620 18:44:19.388818 18002 heartbeater.cc:503] Master 127.15.102.254:46117 was elected leader, sending a full tablet report...
I20260620 18:44:19.391547 18024 ts_tablet_manager.cc:1428] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Time spent starting tablet: real 0.024s user 0.008s sys 0.011s
W20260620 18:44:19.425235 17725 tablet.cc:2369] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:44:19.434412 17864 tablet.cc:2369] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:44:19.455325 18003 tablet.cc:2369] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:19.577670 18028 raft_consensus.cc:482] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:19.577960 18028 raft_consensus.cc:504] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.579555 18028 leader_election.cc:288] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345), 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285)
I20260620 18:44:19.585985 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.579838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:45285 (local address 127.15.102.195:50127)
0620 18:44:19.580068 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.580093 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.580234 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.580866 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.580871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.580989 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:19.581182 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.581195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.582374 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.582379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.583957 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.583986 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.585178 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.585185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.585397 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.585430 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.585526 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.585659 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":74,"spinlock_wait_cycles":26496}
I20260620 18:44:19.586082 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.580274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:50127 (local address 127.15.102.194:45285)
0620 18:44:19.580513 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.580520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.580542 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:44:19.580631 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.580635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.580711 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.580815 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:19.581320 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.582235 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.584124 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.585019 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.585194 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.585267 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.585709 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.585835 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:44:19.586292 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.580108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:56037 (local address 127.15.102.193:35345)
0620 18:44:19.580307 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.580313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.580885 (+ 572us) server_negotiation.cc:407] Connection header received
0620 18:44:19.581044 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.581049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.581123 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.581229 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:19.581807 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.582589 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.584411 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.585365 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.585556 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.585626 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.585932 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.586064 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:44:19.586922 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.579996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:35345 (local address 127.15.102.195:56037)
0620 18:44:19.580769 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.580794 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.580911 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.581266 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.581273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.581299 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:19.581662 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.581672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.582731 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.582735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.584264 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.584278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.585585 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.585592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.585756 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.585792 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.586492 (+ 700us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.586635 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":1,"thread_start_us":472,"threads_started":1}
I20260620 18:44:19.587543 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476255c253364b23956d8ba323c0a686" candidate_uuid: "809d6c5efa0349f684708986558bad8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" is_pre_election: true
I20260620 18:44:19.587903 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476255c253364b23956d8ba323c0a686" candidate_uuid: "809d6c5efa0349f684708986558bad8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eb54b34c054558ac49176d4a270275" is_pre_election: true
I20260620 18:44:19.588037 17818 raft_consensus.cc:2454] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809d6c5efa0349f684708986558bad8e in term 0.
I20260620 18:44:19.588346 17678 raft_consensus.cc:2454] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809d6c5efa0349f684708986558bad8e in term 0.
I20260620 18:44:19.588721 17885 leader_election.cc:302] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8cb03b7c104cc4bf844d3e9d9f33f2, 809d6c5efa0349f684708986558bad8e; no voters:
I20260620 18:44:19.589133 18028 raft_consensus.cc:2781] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:19.589298 18028 raft_consensus.cc:482] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:19.589407 18028 raft_consensus.cc:3037] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:19.591572 18028 raft_consensus.cc:504] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.592166 18028 leader_election.cc:288] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [CANDIDATE]: Term 1 election: Requested vote from peers f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345), 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285)
I20260620 18:44:19.592609 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476255c253364b23956d8ba323c0a686" candidate_uuid: "809d6c5efa0349f684708986558bad8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eb54b34c054558ac49176d4a270275"
I20260620 18:44:19.592638 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476255c253364b23956d8ba323c0a686" candidate_uuid: "809d6c5efa0349f684708986558bad8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2"
I20260620 18:44:19.592798 17678 raft_consensus.cc:3037] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:19.592821 17818 raft_consensus.cc:3037] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:19.594976 17818 raft_consensus.cc:2454] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809d6c5efa0349f684708986558bad8e in term 1.
I20260620 18:44:19.595093 17678 raft_consensus.cc:2454] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809d6c5efa0349f684708986558bad8e in term 1.
I20260620 18:44:19.595430 17885 leader_election.cc:302] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8cb03b7c104cc4bf844d3e9d9f33f2, 809d6c5efa0349f684708986558bad8e; no voters:
I20260620 18:44:19.595712 18028 raft_consensus.cc:2781] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:19.596155 18028 raft_consensus.cc:686] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 1 LEADER]: Becoming Leader. State: Replica: 809d6c5efa0349f684708986558bad8e, State: Running, Role: LEADER
I20260620 18:44:19.596647 18028 consensus_queue.cc:237] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } }
I20260620 18:44:19.601424 17515 catalog_manager.cc:5495] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d6c5efa0349f684708986558bad8e (127.15.102.195). New cstate: current_term: 1 leader_uuid: "809d6c5efa0349f684708986558bad8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:44:19 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981057, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:19.706965 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.693601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46117 (local address 127.0.0.1:39684)
0620 18:44:19.694114 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.694129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.694245 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.694535 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.694538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.695207 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:19.695510 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.695519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.696532 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.696535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.697507 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.697517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.698348 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.698353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.698519 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.699221 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:19.699298 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:19.701397 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:19.704742 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:19.704751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:19.704757 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:19.705181 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:19.705500 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:19.705504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:19.705507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:19.705634 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:19.706110 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:19.706122 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:19.706314 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.706482 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.706663 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":270,"threads_started":1}
I20260620 18:44:19.707165 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.693577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.15.102.254:46117)
0620 18:44:19.693750 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.693755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.694213 (+ 458us) server_negotiation.cc:407] Connection header received
0620 18:44:19.694348 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.694352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.694408 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.694515 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:19.695662 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.696415 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.697617 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.698185 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.698281 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.701569 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:19.701574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:19.701577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:19.701604 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:19.704624 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:19.705281 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:19.705285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:19.705287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:19.705407 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:19.705726 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:19.705729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:19.705731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:19.705984 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:19.706105 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.706628 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.706792 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:44:19.712179 17515 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:39684:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:44:19.713343 17515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:19.720547 17793 tablet_service.cc:1467] Processing CreateTablet for tablet d9f0b4a91469410282d11ee006f8f627 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3aca067704084130a265a773c09b7f45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:44:19.720396 17654 tablet_service.cc:1467] Processing CreateTablet for tablet d9f0b4a91469410282d11ee006f8f627 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3aca067704084130a265a773c09b7f45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:44:19.720767 17932 tablet_service.cc:1467] Processing CreateTablet for tablet d9f0b4a91469410282d11ee006f8f627 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3aca067704084130a265a773c09b7f45]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:44:19.721364 17793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f0b4a91469410282d11ee006f8f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:19.721439 17654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f0b4a91469410282d11ee006f8f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:19.721542 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f0b4a91469410282d11ee006f8f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:19.727303 18021 tablet_bootstrap.cc:492] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Bootstrap starting.
I20260620 18:44:19.728729 18021 tablet_bootstrap.cc:654] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:19.728777 18024 tablet_bootstrap.cc:492] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: Bootstrap starting.
I20260620 18:44:19.729348 18022 tablet_bootstrap.cc:492] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: Bootstrap starting.
I20260620 18:44:19.730548 18024 tablet_bootstrap.cc:654] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:19.731220 18021 tablet_bootstrap.cc:492] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: No bootstrap required, opened a new log
I20260620 18:44:19.731289 18022 tablet_bootstrap.cc:654] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:19.731444 18021 ts_tablet_manager.cc:1397] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260620 18:44:19.732293 18021 raft_consensus.cc:348] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.732528 18021 raft_consensus.cc:374] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:19.732621 18021 raft_consensus.cc:729] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8cb03b7c104cc4bf844d3e9d9f33f2, State: Initialized, Role: FOLLOWER
I20260620 18:44:19.732865 18021 consensus_queue.cc:260] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.733207 18024 tablet_bootstrap.cc:492] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: No bootstrap required, opened a new log
I20260620 18:44:19.733399 18024 ts_tablet_manager.cc:1397] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260620 18:44:19.733310 18027 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.733669 18027 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:19.734647 18021 ts_tablet_manager.cc:1428] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 18:44:19.734459 18024 raft_consensus.cc:348] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.734792 18024 raft_consensus.cc:374] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:19.734910 18024 raft_consensus.cc:729] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eb54b34c054558ac49176d4a270275, State: Initialized, Role: FOLLOWER
I20260620 18:44:19.735212 18024 consensus_queue.cc:260] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.735826 18029 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.736145 18024 ts_tablet_manager.cc:1428] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 18:44:19.736171 18029 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:19.737846 18022 tablet_bootstrap.cc:492] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: No bootstrap required, opened a new log
I20260620 18:44:19.738018 18022 ts_tablet_manager.cc:1397] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260620 18:44:19.739567 18022 raft_consensus.cc:348] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.739766 18022 raft_consensus.cc:374] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:19.739861 18022 raft_consensus.cc:729] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d6c5efa0349f684708986558bad8e, State: Initialized, Role: FOLLOWER
I20260620 18:44:19.740077 18022 consensus_queue.cc:260] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.740450 18028 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.740688 18028 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:19.741138 18022 ts_tablet_manager.cc:1428] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:19.758442 18027 raft_consensus.cc:482] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:19.758715 18027 raft_consensus.cc:504] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.760594 18027 leader_election.cc:288] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345), 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445)
I20260620 18:44:19.767081 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.760975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:48391 (local address 127.15.102.193:35345)
0620 18:44:19.761152 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.761156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.761733 (+ 577us) server_negotiation.cc:407] Connection header received
0620 18:44:19.762002 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.762011 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.762092 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.762156 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:19.762843 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.763708 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.765414 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.766157 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.766342 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.766425 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.766692 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.766811 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:44:19.767367 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.760927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:35345 (local address 127.15.102.194:48391)
0620 18:44:19.761612 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.761635 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.761772 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.762343 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.762346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.762367 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:19.762701 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.762710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.763839 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.763845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.765257 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.765268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.766362 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.766369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.766521 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.766543 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.766667 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.766799 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":385,"threads_started":1}
I20260620 18:44:19.767378 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.761930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:46105 (local address 127.15.102.195:33445)
0620 18:44:19.762132 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.762140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.762409 (+ 269us) server_negotiation.cc:407] Connection header received
0620 18:44:19.762647 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.762656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.762709 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.762802 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:19.763382 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.764067 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.765675 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.766435 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.766618 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.766690 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.766957 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.767075 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:44:19.767575 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.761738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:33445 (local address 127.15.102.194:46105)
0620 18:44:19.762302 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.762315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.762495 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.762814 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.762818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.762874 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:19.763175 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.763182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.764202 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.764207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.765530 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.765540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.766646 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.766653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.766800 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.766822 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.766938 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.767056 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":270,"threads_started":1}
I20260620 18:44:19.768250 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f0b4a91469410282d11ee006f8f627" candidate_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eb54b34c054558ac49176d4a270275" is_pre_election: true
I20260620 18:44:19.768514 17678 raft_consensus.cc:2454] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8cb03b7c104cc4bf844d3e9d9f33f2 in term 0.
I20260620 18:44:19.768378 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f0b4a91469410282d11ee006f8f627" candidate_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809d6c5efa0349f684708986558bad8e" is_pre_election: true
I20260620 18:44:19.768792 17957 raft_consensus.cc:2454] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8cb03b7c104cc4bf844d3e9d9f33f2 in term 0.
I20260620 18:44:19.769085 17747 leader_election.cc:302] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8cb03b7c104cc4bf844d3e9d9f33f2, f5eb54b34c054558ac49176d4a270275; no voters:
I20260620 18:44:19.769517 18027 raft_consensus.cc:2781] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:19.769618 18027 raft_consensus.cc:482] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:19.769757 18027 raft_consensus.cc:3037] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:19.772140 18027 raft_consensus.cc:504] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.772691 18027 leader_election.cc:288] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [CANDIDATE]: Term 1 election: Requested vote from peers f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345), 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445)
I20260620 18:44:19.773200 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f0b4a91469410282d11ee006f8f627" candidate_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eb54b34c054558ac49176d4a270275"
I20260620 18:44:19.773254 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f0b4a91469410282d11ee006f8f627" candidate_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809d6c5efa0349f684708986558bad8e"
I20260620 18:44:19.773414 17678 raft_consensus.cc:3037] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:19.773495 17957 raft_consensus.cc:3037] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:19.775432 17678 raft_consensus.cc:2454] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8cb03b7c104cc4bf844d3e9d9f33f2 in term 1.
I20260620 18:44:19.775671 17957 raft_consensus.cc:2454] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8cb03b7c104cc4bf844d3e9d9f33f2 in term 1.
I20260620 18:44:19.775844 17747 leader_election.cc:302] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8cb03b7c104cc4bf844d3e9d9f33f2, f5eb54b34c054558ac49176d4a270275; no voters:
I20260620 18:44:19.776127 18027 raft_consensus.cc:2781] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:19.776544 18027 raft_consensus.cc:686] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 LEADER]: Becoming Leader. State: Replica: 4c8cb03b7c104cc4bf844d3e9d9f33f2, State: Running, Role: LEADER
I20260620 18:44:19.777002 18027 consensus_queue.cc:237] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:19.780117 18044 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: TxnStatusTablet state changed. Reason: New leader 4c8cb03b7c104cc4bf844d3e9d9f33f2. Latest consensus state: current_term: 1 leader_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.780452 18044 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:44:19.781224 18046 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:44:19.781495 18046 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:44:19.782022 17515 catalog_manager.cc:5495] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194). New cstate: current_term: 1 leader_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } health_report { overall_health: UNKNOWN } } }
I20260620 18:44:19.813894 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.807460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:45285 (local address 127.0.0.1:54778)
0620 18:44:19.807690 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.807716 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.807835 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.808346 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.808351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.808361 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:44:19.808514 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.808521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.810029 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.810033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.811031 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.811042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.812044 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.812048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.812197 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.812253 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:44:19.813372 (+ 1119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:44:19.813392 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.813462 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.813578 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:44:19.813932 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.807528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.15.102.194:45285)
0620 18:44:19.807721 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.807727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.807774 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:44:19.808028 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.808034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.808140 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.808305 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:19.808681 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.809871 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.811165 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.811844 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.811992 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.812444 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:19.813223 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:19.813368 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.813510 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.813597 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:44:19.821755 17957 raft_consensus.cc:1261] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Refusing update from remote peer 4c8cb03b7c104cc4bf844d3e9d9f33f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:19.821949 17678 raft_consensus.cc:1261] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Refusing update from remote peer 4c8cb03b7c104cc4bf844d3e9d9f33f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:19.822736 18044 consensus_queue.cc:1035] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:19.823115 18027 consensus_queue.cc:1035] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:19.830451 18028 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: TxnStatusTablet state changed. Reason: New leader 4c8cb03b7c104cc4bf844d3e9d9f33f2. Latest consensus state: current_term: 1 leader_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.830833 18028 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:19.835664 18029 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: TxnStatusTablet state changed. Reason: New leader 4c8cb03b7c104cc4bf844d3e9d9f33f2. Latest consensus state: current_term: 1 leader_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.835968 18029 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:19.836841 18050 mvcc.cc:205] Tried to move back new op lower bound from 7298994421020790784 to 7298994420850184192. Current Snapshot: MvccSnapshot[applied={T|T < 7298994421020790784}]
I20260620 18:44:19.839136 18044 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.839422 18044 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:44:19.839607 18044 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.839823 18044 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:44:19.839850 18028 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.840109 18028 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P 809d6c5efa0349f684708986558bad8e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:19.841377 18049 mvcc.cc:205] Tried to move back new op lower bound from 7298994421020790784 to 7298994420850184192. Current Snapshot: MvccSnapshot[applied={T|T < 7298994421020790784 or (T in {7298994421020790784})}]
I20260620 18:44:19.855130 18029 tablet_replica.cc:439] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } } }
I20260620 18:44:19.855417 18029 tablet_replica.cc:442] T d9f0b4a91469410282d11ee006f8f627 P f5eb54b34c054558ac49176d4a270275: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:44:19.859405 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.851389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.15.102.195:33445)
0620 18:44:19.851545 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.851551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.851731 (+ 180us) server_negotiation.cc:407] Connection header received
0620 18:44:19.852087 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.852093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.852189 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.852313 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:19.853448 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.854661 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.857182 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.857590 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.857731 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.858254 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:19.858781 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:19.858954 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.859010 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.859090 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 18:44:19.874917 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.869160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:45285 (local address 127.15.102.195:50467)
0620 18:44:19.869719 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.869735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.869845 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.870177 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.870180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.870189 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:44:19.870444 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.870451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.871888 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.871892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.872925 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.872936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.873891 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.873895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.874010 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.874059 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:44:19.874456 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:44:19.874463 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.874521 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.874613 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":286,"threads_started":1}
I20260620 18:44:19.874927 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.869278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:50467 (local address 127.15.102.194:45285)
0620 18:44:19.869488 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.869493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.869812 (+ 319us) server_negotiation.cc:407] Connection header received
0620 18:44:19.869972 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.869979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.870066 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.870171 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:19.870573 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.871721 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.873057 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.873703 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.873832 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.874235 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:19.874360 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:19.874460 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.874565 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.874641 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:44:19.884711 17678 raft_consensus.cc:1261] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Refusing update from remote peer 809d6c5efa0349f684708986558bad8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:19.885259 17818 raft_consensus.cc:1261] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Refusing update from remote peer 809d6c5efa0349f684708986558bad8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:19.885815 18031 consensus_queue.cc:1035] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:19.886417 18031 consensus_queue.cc:1035] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:19.931736 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.924762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:60545 (local address 127.15.102.195:33445)
0620 18:44:19.924944 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:19.924951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:19.925589 (+ 638us) server_negotiation.cc:407] Connection header received
0620 18:44:19.925786 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:19.925792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:19.925881 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:19.925969 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:19.926722 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.927510 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.929680 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.930373 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.930466 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.931099 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:19.931195 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:19.931338 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:19.931466 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:44:19.931525 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 18:44:19.932020 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:19.924783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:33445 (local address 127.15.102.194:60545)
0620 18:44:19.925481 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:19.925501 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:19.925643 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:19.926144 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:19.926149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:19.926163 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:44:19.926520 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.926532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.927633 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.927639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.929517 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:19.929532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:19.930661 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:19.930668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:19.930854 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:19.930930 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:44:19.931339 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:44:19.931351 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:44:19.931442 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:44:19.931588 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":977,"threads_started":1}
I20260620 18:44:19.970521 17517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:19.970801 17517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:19.974725 17517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: Sending DeleteTablet for 3 replicas of tablet 476255c253364b23956d8ba323c0a686
I20260620 18:44:19.976050 17793 tablet_service.cc:1514] Processing DeleteTablet for tablet 476255c253364b23956d8ba323c0a686 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:19 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:54766
I20260620 18:44:19.976018 17654 tablet_service.cc:1514] Processing DeleteTablet for tablet 476255c253364b23956d8ba323c0a686 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:19 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260620 18:44:19.976915 17932 tablet_service.cc:1514] Processing DeleteTablet for tablet 476255c253364b23956d8ba323c0a686 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:19 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:38700
I20260620 18:44:19.979947 17517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:19.980602 17517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:19.987941 17792 tablet_service.cc:1467] Processing CreateTablet for tablet b218a37b01204fbbbde918a654288f63 (DEFAULT_TABLE table=test-table [id=23847ba858124aeb9473ea9d6a43ab0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:19.988485 17792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b218a37b01204fbbbde918a654288f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:19.992193 17653 tablet_service.cc:1467] Processing CreateTablet for tablet b218a37b01204fbbbde918a654288f63 (DEFAULT_TABLE table=test-table [id=23847ba858124aeb9473ea9d6a43ab0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:19.992705 17653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b218a37b01204fbbbde918a654288f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:19.995992 17931 tablet_service.cc:1467] Processing CreateTablet for tablet b218a37b01204fbbbde918a654288f63 (DEFAULT_TABLE table=test-table [id=23847ba858124aeb9473ea9d6a43ab0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:19.996524 17931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b218a37b01204fbbbde918a654288f63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:20.005079 18022 tablet_bootstrap.cc:492] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: Bootstrap starting.
I20260620 18:44:20.005673 18024 tablet_bootstrap.cc:492] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: Bootstrap starting.
I20260620 18:44:20.006824 18022 tablet_bootstrap.cc:654] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:20.007311 18024 tablet_bootstrap.cc:654] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:20.012837 18022 tablet_bootstrap.cc:492] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: No bootstrap required, opened a new log
I20260620 18:44:20.013003 18022 ts_tablet_manager.cc:1397] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260620 18:44:20.013815 18022 raft_consensus.cc:348] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.014072 18022 raft_consensus.cc:374] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:20.014148 18022 raft_consensus.cc:729] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d6c5efa0349f684708986558bad8e, State: Initialized, Role: FOLLOWER
I20260620 18:44:20.014457 18022 consensus_queue.cc:260] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.015156 18022 ts_tablet_manager.cc:1428] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:44:20.016012 18021 tablet_bootstrap.cc:492] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Bootstrap starting.
I20260620 18:44:20.017769 18021 tablet_bootstrap.cc:654] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:20.022882 18073 tablet_replica.cc:330] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: stopping tablet replica
I20260620 18:44:20.024562 18073 raft_consensus.cc:2229] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:20.025336 18073 raft_consensus.cc:2258] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:20.026124 18072 tablet_replica.cc:330] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: stopping tablet replica
I20260620 18:44:20.028002 18073 ts_tablet_manager.cc:1905] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:20.031064 18072 raft_consensus.cc:2229] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:20.031620 18072 raft_consensus.cc:2258] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:20.032493 18071 tablet_replica.cc:330] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: stopping tablet replica
I20260620 18:44:20.036667 18072 ts_tablet_manager.cc:1905] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:20.036944 18021 tablet_bootstrap.cc:492] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: No bootstrap required, opened a new log
I20260620 18:44:20.037142 18021 ts_tablet_manager.cc:1397] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.008s
I20260620 18:44:20.037325 18071 raft_consensus.cc:2229] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:20.037716 18071 raft_consensus.cc:2258] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:20.037985 18021 raft_consensus.cc:348] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.038275 18021 raft_consensus.cc:374] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:20.038380 18021 raft_consensus.cc:729] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8cb03b7c104cc4bf844d3e9d9f33f2, State: Initialized, Role: FOLLOWER
I20260620 18:44:20.038935 18021 consensus_queue.cc:260] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.039525 18021 ts_tablet_manager.cc:1428] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:44:20.040745 18024 tablet_bootstrap.cc:492] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: No bootstrap required, opened a new log
I20260620 18:44:20.040910 18024 ts_tablet_manager.cc:1397] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.009s
I20260620 18:44:20.041662 18024 raft_consensus.cc:348] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.041901 18024 raft_consensus.cc:374] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:20.041976 18024 raft_consensus.cc:729] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eb54b34c054558ac49176d4a270275, State: Initialized, Role: FOLLOWER
I20260620 18:44:20.042241 18024 consensus_queue.cc:260] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.043479 18024 ts_tablet_manager.cc:1428] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:44:20.046751 18071 ts_tablet_manager.cc:1905] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:20.057643 18072 ts_tablet_manager.cc:1918] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:20.057837 18072 log.cc:1196] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/476255c253364b23956d8ba323c0a686
I20260620 18:44:20.058331 18072 ts_tablet_manager.cc:1939] T 476255c253364b23956d8ba323c0a686 P f5eb54b34c054558ac49176d4a270275: Deleting consensus metadata
I20260620 18:44:20.060292 17504 catalog_manager.cc:4841] TS f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345): tablet 476255c253364b23956d8ba323c0a686 (table test-table [id=ab362e91583b4c078ad4384cfb046b9b]) successfully deleted
I20260620 18:44:20.066215 18071 ts_tablet_manager.cc:1918] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:20.066421 18071 log.cc:1196] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/476255c253364b23956d8ba323c0a686
I20260620 18:44:20.067003 18071 ts_tablet_manager.cc:1939] T 476255c253364b23956d8ba323c0a686 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting consensus metadata
I20260620 18:44:20.067390 18073 ts_tablet_manager.cc:1918] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:20.067615 18073 log.cc:1196] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/476255c253364b23956d8ba323c0a686
I20260620 18:44:20.068287 18073 ts_tablet_manager.cc:1939] T 476255c253364b23956d8ba323c0a686 P 809d6c5efa0349f684708986558bad8e: Deleting consensus metadata
I20260620 18:44:20.068981 17503 catalog_manager.cc:4841] TS 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285): tablet 476255c253364b23956d8ba323c0a686 (table test-table [id=ab362e91583b4c078ad4384cfb046b9b]) successfully deleted
I20260620 18:44:20.070086 17504 catalog_manager.cc:4841] TS 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445): tablet 476255c253364b23956d8ba323c0a686 (table test-table [id=ab362e91583b4c078ad4384cfb046b9b]) successfully deleted
I20260620 18:44:20.074584 18029 raft_consensus.cc:482] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:20.074836 18029 raft_consensus.cc:504] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.076740 18029 leader_election.cc:288] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285), 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445)
I20260620 18:44:20.098155 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.077589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:55425 (local address 127.15.102.194:45285)
0620 18:44:20.077778 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:20.077785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:20.089903 (+ 12118us) server_negotiation.cc:407] Connection header received
0620 18:44:20.090393 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:20.090400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:20.090478 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:20.090540 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:20.091357 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.092547 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.095257 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.096728 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.096877 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.096959 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:20.097649 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 18:44:20.097799 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:44:20.100018 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.077751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:42457 (local address 127.15.102.195:33445)
0620 18:44:20.077946 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:20.077951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:20.090143 (+ 12192us) server_negotiation.cc:407] Connection header received
0620 18:44:20.090561 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:20.090567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:20.090642 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:20.090714 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:20.093100 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.094255 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.096989 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.098426 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.098582 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.098660 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:20.099555 (+ 895us) server_negotiation.cc:299] Negotiation successful
0620 18:44:20.099704 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:44:20.100153 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.077249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:45285 (local address 127.15.102.193:55425)
0620 18:44:20.089758 (+ 12509us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:20.089788 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:20.090211 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:20.090724 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:20.090729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:20.090753 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:20.091138 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:20.091149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.092730 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.092735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:20.095051 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:20.095065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.097277 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.097284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:20.097464 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.097498 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:44:20.099598 (+ 2100us) client_negotiation.cc:240] Negotiation successful
0620 18:44:20.099751 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12351,"thread_start_us":12682,"threads_started":1}
I20260620 18:44:20.100780 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b218a37b01204fbbbde918a654288f63" candidate_uuid: "f5eb54b34c054558ac49176d4a270275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" is_pre_election: true
I20260620 18:44:20.101075 17818 raft_consensus.cc:2454] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5eb54b34c054558ac49176d4a270275 in term 0.
I20260620 18:44:20.101219 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.077275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:33445 (local address 127.15.102.193:42457)
0620 18:44:20.090037 (+ 12762us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:20.090058 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:20.090394 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:20.090922 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:20.090926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:20.090950 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:20.091431 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:20.091447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.094491 (+ 3044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.094495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:20.096775 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:20.096792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.099171 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.099177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:20.099370 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.099400 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:44:20.100540 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0620 18:44:20.100681 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12610,"thread_start_us":12881,"threads_started":1}
I20260620 18:44:20.101611 17606 leader_election.cc:302] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8cb03b7c104cc4bf844d3e9d9f33f2, f5eb54b34c054558ac49176d4a270275; no voters:
I20260620 18:44:20.101958 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b218a37b01204fbbbde918a654288f63" candidate_uuid: "f5eb54b34c054558ac49176d4a270275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809d6c5efa0349f684708986558bad8e" is_pre_election: true
I20260620 18:44:20.102067 18029 raft_consensus.cc:2781] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:20.102180 18029 raft_consensus.cc:482] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:20.102224 17957 raft_consensus.cc:2454] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5eb54b34c054558ac49176d4a270275 in term 0.
I20260620 18:44:20.102257 18029 raft_consensus.cc:3037] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:20.105490 18029 raft_consensus.cc:504] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.106202 18029 leader_election.cc:288] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [CANDIDATE]: Term 1 election: Requested vote from peers 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285), 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445)
I20260620 18:44:20.106741 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b218a37b01204fbbbde918a654288f63" candidate_uuid: "f5eb54b34c054558ac49176d4a270275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2"
I20260620 18:44:20.106815 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b218a37b01204fbbbde918a654288f63" candidate_uuid: "f5eb54b34c054558ac49176d4a270275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809d6c5efa0349f684708986558bad8e"
I20260620 18:44:20.106987 17818 raft_consensus.cc:3037] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:20.107028 17957 raft_consensus.cc:3037] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:20.110159 17957 raft_consensus.cc:2454] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5eb54b34c054558ac49176d4a270275 in term 1.
I20260620 18:44:20.110169 17818 raft_consensus.cc:2454] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5eb54b34c054558ac49176d4a270275 in term 1.
I20260620 18:44:20.110651 17607 leader_election.cc:302] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809d6c5efa0349f684708986558bad8e, f5eb54b34c054558ac49176d4a270275; no voters:
I20260620 18:44:20.111277 18029 raft_consensus.cc:2781] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:20.112839 18029 raft_consensus.cc:686] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 1 LEADER]: Becoming Leader. State: Replica: f5eb54b34c054558ac49176d4a270275, State: Running, Role: LEADER
I20260620 18:44:20.113409 18029 consensus_queue.cc:237] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:20.117457 17517 catalog_manager.cc:5495] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5eb54b34c054558ac49176d4a270275 (127.15.102.193). New cstate: current_term: 1 leader_uuid: "f5eb54b34c054558ac49176d4a270275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } health_report { overall_health: UNKNOWN } } }
I20260620 18:44:20.160429 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.152593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.15.102.193:35345)
0620 18:44:20.152749 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:20.152755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:20.152807 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:44:20.153043 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:20.153049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:20.153167 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:20.153328 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:20.153924 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.155038 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.157103 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.157736 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.157900 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.158491 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:20.159304 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:20.159452 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:20.160054 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 18:44:20.160135 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:44:20.196035 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.181115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:41277 (local address 127.15.102.194:45285)
0620 18:44:20.181315 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:20.181321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:20.188290 (+ 6969us) server_negotiation.cc:407] Connection header received
0620 18:44:20.188782 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:20.188789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:20.188892 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:20.188960 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:20.189792 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.190953 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.193683 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.194346 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.194454 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.195195 (+ 741us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:20.195361 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:20.195427 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:20.195649 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 18:44:20.195733 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:44:20.197103 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.181464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:45285 (local address 127.15.102.193:41277)
0620 18:44:20.188153 (+ 6689us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:20.188183 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:20.188601 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:20.189168 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:20.189173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:20.189185 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:44:20.189571 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:20.189585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.191145 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.191150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:20.193515 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:20.193533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.194669 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.194676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:20.194897 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.194972 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:44:20.195514 (+ 542us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:44:20.195524 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:44:20.196505 (+ 981us) client_negotiation.cc:240] Negotiation successful
0620 18:44:20.196652 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6452,"thread_start_us":6846,"threads_started":1}
I20260620 18:44:20.210803 17818 raft_consensus.cc:1261] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Refusing update from remote peer f5eb54b34c054558ac49176d4a270275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:20.212554 18079 consensus_queue.cc:1035] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:20.215296 17957 raft_consensus.cc:1261] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Refusing update from remote peer f5eb54b34c054558ac49176d4a270275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:20.216148 18079 consensus_queue.cc:1035] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:20.219780 18051 mvcc.cc:205] Tried to move back new op lower bound from 7298994422615080960 to 7298994422229590016. Current Snapshot: MvccSnapshot[applied={T|T < 7298994422615080960}]
I20260620 18:44:20.270788 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.252046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:50283 (local address 127.15.102.193:35345)
0620 18:44:20.252216 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:20.252222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:20.252238 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:44:20.252298 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:20.252304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:20.252378 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:20.252461 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:20.252997 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.254092 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.258445 (+ 4353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.259168 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.259316 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.266878 (+ 7562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:20.267019 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:20.267100 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:20.270389 (+ 3289us) server_negotiation.cc:299] Negotiation successful
0620 18:44:20.270477 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:44:20.271548 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:20.251457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:35345 (local address 127.15.102.194:50283)
0620 18:44:20.251709 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:20.251734 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:20.251844 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:20.252613 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:20.252618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:20.252633 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:44:20.252830 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:20.252843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.254254 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.254263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:20.255859 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:20.255892 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:20.266373 (+ 10481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:20.266382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:20.266617 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:20.266678 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:44:20.267373 (+ 695us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:44:20.267385 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:44:20.267473 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 18:44:20.267661 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
W20260620 18:44:20.273053 18051 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:44:21.912050 17517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:21.912254 17517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:21.916190 17517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: Sending DeleteTablet for 3 replicas of tablet b218a37b01204fbbbde918a654288f63
I20260620 18:44:21.916872 17654 tablet_service.cc:1514] Processing DeleteTablet for tablet b218a37b01204fbbbde918a654288f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:21 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260620 18:44:21.916967 17793 tablet_service.cc:1514] Processing DeleteTablet for tablet b218a37b01204fbbbde918a654288f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:21 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:54766
I20260620 18:44:21.917318 17932 tablet_service.cc:1514] Processing DeleteTablet for tablet b218a37b01204fbbbde918a654288f63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:21 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:38700
I20260620 18:44:21.917865 18110 tablet_replica.cc:330] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: stopping tablet replica
I20260620 18:44:21.918066 18110 raft_consensus.cc:2229] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:21.918236 18111 tablet_replica.cc:330] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: stopping tablet replica
I20260620 18:44:21.918455 18111 raft_consensus.cc:2229] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:21.919095 18110 raft_consensus.cc:2258] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:21.919152 18111 raft_consensus.cc:2258] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:21.920131 18111 ts_tablet_manager.cc:1905] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:21.920121 18110 ts_tablet_manager.cc:1905] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:21.926640 18111 ts_tablet_manager.cc:1918] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:44:21.926640 18110 ts_tablet_manager.cc:1918] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:44:21.926774 18110 log.cc:1196] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/b218a37b01204fbbbde918a654288f63
I20260620 18:44:21.926900 18111 log.cc:1196] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/b218a37b01204fbbbde918a654288f63
I20260620 18:44:21.927260 18110 ts_tablet_manager.cc:1939] T b218a37b01204fbbbde918a654288f63 P f5eb54b34c054558ac49176d4a270275: Deleting consensus metadata
I20260620 18:44:21.927503 18111 ts_tablet_manager.cc:1939] T b218a37b01204fbbbde918a654288f63 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting consensus metadata
I20260620 18:44:21.928408 17504 catalog_manager.cc:4841] TS f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345): tablet b218a37b01204fbbbde918a654288f63 (table test-table [id=23847ba858124aeb9473ea9d6a43ab0f]) successfully deleted
I20260620 18:44:21.928491 18112 tablet_replica.cc:330] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: stopping tablet replica
I20260620 18:44:21.928539 17503 catalog_manager.cc:4841] TS 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285): tablet b218a37b01204fbbbde918a654288f63 (table test-table [id=23847ba858124aeb9473ea9d6a43ab0f]) successfully deleted
I20260620 18:44:21.928714 18112 raft_consensus.cc:2229] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:21.928936 18112 raft_consensus.cc:2258] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:21.929792 18112 ts_tablet_manager.cc:1905] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:21.934440 18112 ts_tablet_manager.cc:1918] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:44:21.934590 18112 log.cc:1196] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/b218a37b01204fbbbde918a654288f63
I20260620 18:44:21.935030 18112 ts_tablet_manager.cc:1939] T b218a37b01204fbbbde918a654288f63 P 809d6c5efa0349f684708986558bad8e: Deleting consensus metadata
I20260620 18:44:21.935997 17504 catalog_manager.cc:4841] TS 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445): tablet b218a37b01204fbbbde918a654288f63 (table test-table [id=23847ba858124aeb9473ea9d6a43ab0f]) successfully deleted
Jun 20 18:44:21 dist-test-slave-zfp8 krb5kdc[17472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:44:21 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:21.973317 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:21.955538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.15.102.254:46117)
0620 18:44:21.956098 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:21.956103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:21.956125 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:44:21.956291 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:21.956295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:21.956354 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:21.956450 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:21.957767 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:21.958536 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:21.959827 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:21.960226 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:21.960355 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:21.968518 (+ 8163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:21.968523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:21.968526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:21.968561 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:21.971059 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:21.971724 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:21.971728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:21.971731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:21.971844 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:21.972134 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:21.972137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:21.972139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:21.972388 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:21.972577 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:21.972807 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:44:21.972959 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
I20260620 18:44:21.976718 17517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:21.977397 17517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:21.983527 17654 tablet_service.cc:1467] Processing CreateTablet for tablet 8571ab1d4ee942fda50e42bbfd748fc8 (DEFAULT_TABLE table=test-table [id=41d3012f0e1d4313b157600f84d16cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:21.983657 17793 tablet_service.cc:1467] Processing CreateTablet for tablet 8571ab1d4ee942fda50e42bbfd748fc8 (DEFAULT_TABLE table=test-table [id=41d3012f0e1d4313b157600f84d16cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:21.983964 17654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8571ab1d4ee942fda50e42bbfd748fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:21.984025 17793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8571ab1d4ee942fda50e42bbfd748fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:21.987617 17932 tablet_service.cc:1467] Processing CreateTablet for tablet 8571ab1d4ee942fda50e42bbfd748fc8 (DEFAULT_TABLE table=test-table [id=41d3012f0e1d4313b157600f84d16cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:21.987998 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8571ab1d4ee942fda50e42bbfd748fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:21.991483 18122 tablet_bootstrap.cc:492] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Bootstrap starting.
I20260620 18:44:21.991600 18121 tablet_bootstrap.cc:492] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: Bootstrap starting.
I20260620 18:44:21.992866 18121 tablet_bootstrap.cc:654] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:21.993269 18122 tablet_bootstrap.cc:654] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:21.994922 18124 tablet_bootstrap.cc:492] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: Bootstrap starting.
I20260620 18:44:21.995393 18121 tablet_bootstrap.cc:492] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: No bootstrap required, opened a new log
I20260620 18:44:21.995556 18121 ts_tablet_manager.cc:1397] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:44:21.996155 18122 tablet_bootstrap.cc:492] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: No bootstrap required, opened a new log
I20260620 18:44:21.996289 18122 ts_tablet_manager.cc:1397] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 18:44:21.996317 18121 raft_consensus.cc:348] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:21.996582 18121 raft_consensus.cc:374] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:21.996677 18121 raft_consensus.cc:729] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eb54b34c054558ac49176d4a270275, State: Initialized, Role: FOLLOWER
I20260620 18:44:21.996780 18124 tablet_bootstrap.cc:654] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:21.996927 18121 consensus_queue.cc:260] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:21.997048 18122 raft_consensus.cc:348] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:21.997303 18122 raft_consensus.cc:374] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:21.997406 18122 raft_consensus.cc:729] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8cb03b7c104cc4bf844d3e9d9f33f2, State: Initialized, Role: FOLLOWER
I20260620 18:44:21.997485 18121 ts_tablet_manager.cc:1428] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:44:21.997671 18122 consensus_queue.cc:260] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:21.998345 18122 ts_tablet_manager.cc:1428] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:44:21.999706 18124 tablet_bootstrap.cc:492] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: No bootstrap required, opened a new log
I20260620 18:44:21.999843 18124 ts_tablet_manager.cc:1397] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 18:44:22.000612 18124 raft_consensus.cc:348] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.000864 18124 raft_consensus.cc:374] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:22.000955 18124 raft_consensus.cc:729] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d6c5efa0349f684708986558bad8e, State: Initialized, Role: FOLLOWER
I20260620 18:44:22.001196 18124 consensus_queue.cc:260] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.002012 18124 ts_tablet_manager.cc:1428] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:44:22.194535 18103 raft_consensus.cc:482] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:22.194715 18103 raft_consensus.cc:504] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.195487 18103 leader_election.cc:288] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285), 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445)
I20260620 18:44:22.196040 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8571ab1d4ee942fda50e42bbfd748fc8" candidate_uuid: "f5eb54b34c054558ac49176d4a270275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" is_pre_election: true
I20260620 18:44:22.196102 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8571ab1d4ee942fda50e42bbfd748fc8" candidate_uuid: "f5eb54b34c054558ac49176d4a270275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809d6c5efa0349f684708986558bad8e" is_pre_election: true
I20260620 18:44:22.196269 17818 raft_consensus.cc:2454] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5eb54b34c054558ac49176d4a270275 in term 0.
I20260620 18:44:22.196422 17957 raft_consensus.cc:2454] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5eb54b34c054558ac49176d4a270275 in term 0.
I20260620 18:44:22.196738 17606 leader_election.cc:302] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8cb03b7c104cc4bf844d3e9d9f33f2, f5eb54b34c054558ac49176d4a270275; no voters:
I20260620 18:44:22.197010 18103 raft_consensus.cc:2781] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:22.197113 18103 raft_consensus.cc:482] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:22.197219 18103 raft_consensus.cc:3037] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:22.199504 18103 raft_consensus.cc:504] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.200271 18103 leader_election.cc:288] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [CANDIDATE]: Term 1 election: Requested vote from peers 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285), 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445)
I20260620 18:44:22.200704 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8571ab1d4ee942fda50e42bbfd748fc8" candidate_uuid: "f5eb54b34c054558ac49176d4a270275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2"
I20260620 18:44:22.200779 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8571ab1d4ee942fda50e42bbfd748fc8" candidate_uuid: "f5eb54b34c054558ac49176d4a270275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809d6c5efa0349f684708986558bad8e"
I20260620 18:44:22.200907 17818 raft_consensus.cc:3037] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:22.200992 17957 raft_consensus.cc:3037] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:22.203984 17957 raft_consensus.cc:2454] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5eb54b34c054558ac49176d4a270275 in term 1.
I20260620 18:44:22.203975 17818 raft_consensus.cc:2454] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5eb54b34c054558ac49176d4a270275 in term 1.
I20260620 18:44:22.204356 17606 leader_election.cc:302] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8cb03b7c104cc4bf844d3e9d9f33f2, f5eb54b34c054558ac49176d4a270275; no voters:
I20260620 18:44:22.204631 18103 raft_consensus.cc:2781] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:22.204774 18103 raft_consensus.cc:686] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 1 LEADER]: Becoming Leader. State: Replica: f5eb54b34c054558ac49176d4a270275, State: Running, Role: LEADER
I20260620 18:44:22.205039 18103 consensus_queue.cc:237] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.208031 17515 catalog_manager.cc:5495] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5eb54b34c054558ac49176d4a270275 (127.15.102.193). New cstate: current_term: 1 leader_uuid: "f5eb54b34c054558ac49176d4a270275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } health_report { overall_health: UNKNOWN } } }
I20260620 18:44:22.242110 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:22.236625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.15.102.193:35345)
0620 18:44:22.237199 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:22.237203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:22.237220 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:44:22.237268 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:22.237272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:22.237328 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:22.237430 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:22.237810 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:22.239030 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:22.240394 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:22.240811 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:22.240946 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:22.241384 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:22.241487 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:22.241594 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:22.241735 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:44:22.241809 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":306,"threads_started":1}
I20260620 18:44:22.251019 17957 raft_consensus.cc:1261] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Refusing update from remote peer f5eb54b34c054558ac49176d4a270275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:22.251010 17818 raft_consensus.cc:1261] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Refusing update from remote peer f5eb54b34c054558ac49176d4a270275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:22.251571 18103 consensus_queue.cc:1035] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:22.251820 18088 consensus_queue.cc:1035] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:22.296921 17517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:22.297222 17517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:22.300076 17517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: Sending DeleteTablet for 3 replicas of tablet 8571ab1d4ee942fda50e42bbfd748fc8
I20260620 18:44:22.300853 17654 tablet_service.cc:1514] Processing DeleteTablet for tablet 8571ab1d4ee942fda50e42bbfd748fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:22 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260620 18:44:22.301003 17793 tablet_service.cc:1514] Processing DeleteTablet for tablet 8571ab1d4ee942fda50e42bbfd748fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:22 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:54766
I20260620 18:44:22.301148 18110 tablet_replica.cc:330] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: stopping tablet replica
I20260620 18:44:22.301278 18111 tablet_replica.cc:330] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: stopping tablet replica
I20260620 18:44:22.301334 18110 raft_consensus.cc:2229] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:22.301354 17932 tablet_service.cc:1514] Processing DeleteTablet for tablet 8571ab1d4ee942fda50e42bbfd748fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:22 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:38700
I20260620 18:44:22.301476 18111 raft_consensus.cc:2229] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:22.301616 18112 tablet_replica.cc:330] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: stopping tablet replica
I20260620 18:44:22.301631 18110 raft_consensus.cc:2258] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:22.301664 18111 raft_consensus.cc:2258] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:22.301798 18112 raft_consensus.cc:2229] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:22.301973 18112 raft_consensus.cc:2258] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:22.302574 17517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:44:22.302815 18111 ts_tablet_manager.cc:1905] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:22.303118 18112 ts_tablet_manager.cc:1905] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 18:44:22.303208 17517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:22.308614 18111 ts_tablet_manager.cc:1918] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:22.308774 18111 log.cc:1196] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/8571ab1d4ee942fda50e42bbfd748fc8
I20260620 18:44:22.309006 18112 ts_tablet_manager.cc:1918] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:22.309150 18112 log.cc:1196] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/8571ab1d4ee942fda50e42bbfd748fc8
I20260620 18:44:22.309202 18111 ts_tablet_manager.cc:1939] T 8571ab1d4ee942fda50e42bbfd748fc8 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting consensus metadata
I20260620 18:44:22.309579 18112 ts_tablet_manager.cc:1939] T 8571ab1d4ee942fda50e42bbfd748fc8 P 809d6c5efa0349f684708986558bad8e: Deleting consensus metadata
I20260620 18:44:22.310698 17503 catalog_manager.cc:4841] TS 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285): tablet 8571ab1d4ee942fda50e42bbfd748fc8 (table test-table [id=41d3012f0e1d4313b157600f84d16cf8]) successfully deleted
I20260620 18:44:22.310849 18110 ts_tablet_manager.cc:1905] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:22.311066 17932 tablet_service.cc:1467] Processing CreateTablet for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 (DEFAULT_TABLE table=test-table [id=13a26b9c819147239db03c7433f518b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:22.311216 17793 tablet_service.cc:1467] Processing CreateTablet for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 (DEFAULT_TABLE table=test-table [id=13a26b9c819147239db03c7433f518b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:22.311486 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:22.311637 17793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:22.316428 18124 tablet_bootstrap.cc:492] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: Bootstrap starting.
I20260620 18:44:22.316568 17504 catalog_manager.cc:4841] TS 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445): tablet 8571ab1d4ee942fda50e42bbfd748fc8 (table test-table [id=41d3012f0e1d4313b157600f84d16cf8]) successfully deleted
I20260620 18:44:22.317770 18110 ts_tablet_manager.cc:1918] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:44:22.317911 18110 log.cc:1196] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/8571ab1d4ee942fda50e42bbfd748fc8
I20260620 18:44:22.317997 18124 tablet_bootstrap.cc:654] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:22.318224 18122 tablet_bootstrap.cc:492] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Bootstrap starting.
I20260620 18:44:22.318297 18110 ts_tablet_manager.cc:1939] T 8571ab1d4ee942fda50e42bbfd748fc8 P f5eb54b34c054558ac49176d4a270275: Deleting consensus metadata
I20260620 18:44:22.318800 17654 tablet_service.cc:1467] Processing CreateTablet for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 (DEFAULT_TABLE table=test-table [id=13a26b9c819147239db03c7433f518b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:22.319168 17654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:22.319820 17504 catalog_manager.cc:4841] TS f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345): tablet 8571ab1d4ee942fda50e42bbfd748fc8 (table test-table [id=41d3012f0e1d4313b157600f84d16cf8]) successfully deleted
I20260620 18:44:22.320441 18122 tablet_bootstrap.cc:654] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:22.323015 18122 tablet_bootstrap.cc:492] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: No bootstrap required, opened a new log
I20260620 18:44:22.323217 18122 ts_tablet_manager.cc:1397] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 18:44:22.324138 18121 tablet_bootstrap.cc:492] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: Bootstrap starting.
I20260620 18:44:22.324213 18122 raft_consensus.cc:348] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.324527 18122 raft_consensus.cc:374] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:22.324600 18122 raft_consensus.cc:729] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8cb03b7c104cc4bf844d3e9d9f33f2, State: Initialized, Role: FOLLOWER
I20260620 18:44:22.324846 18122 consensus_queue.cc:260] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.325340 18122 ts_tablet_manager.cc:1428] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:44:22.325752 18121 tablet_bootstrap.cc:654] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:22.325776 18124 tablet_bootstrap.cc:492] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: No bootstrap required, opened a new log
I20260620 18:44:22.325932 18124 ts_tablet_manager.cc:1397] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260620 18:44:22.326658 18124 raft_consensus.cc:348] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.326900 18124 raft_consensus.cc:374] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:22.326984 18124 raft_consensus.cc:729] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d6c5efa0349f684708986558bad8e, State: Initialized, Role: FOLLOWER
I20260620 18:44:22.327289 18124 consensus_queue.cc:260] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.328109 18121 tablet_bootstrap.cc:492] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: No bootstrap required, opened a new log
I20260620 18:44:22.328264 18121 ts_tablet_manager.cc:1397] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 18:44:22.328828 18121 raft_consensus.cc:348] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.329368 18124 ts_tablet_manager.cc:1428] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 18:44:22.328995 18121 raft_consensus.cc:374] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:22.329493 18121 raft_consensus.cc:729] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eb54b34c054558ac49176d4a270275, State: Initialized, Role: FOLLOWER
I20260620 18:44:22.329849 18121 consensus_queue.cc:260] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.330360 18121 ts_tablet_manager.cc:1428] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 18:44:22.332129 18127 raft_consensus.cc:482] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:22.332307 18127 raft_consensus.cc:504] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.332928 18127 leader_election.cc:288] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345), 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285)
I20260620 18:44:22.333387 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995cfb43dca94ab4a5afbdb9e9e82ef0" candidate_uuid: "809d6c5efa0349f684708986558bad8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" is_pre_election: true
I20260620 18:44:22.333561 17818 raft_consensus.cc:2454] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809d6c5efa0349f684708986558bad8e in term 0.
I20260620 18:44:22.333688 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995cfb43dca94ab4a5afbdb9e9e82ef0" candidate_uuid: "809d6c5efa0349f684708986558bad8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eb54b34c054558ac49176d4a270275" is_pre_election: true
I20260620 18:44:22.333891 17678 raft_consensus.cc:2454] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 809d6c5efa0349f684708986558bad8e in term 0.
I20260620 18:44:22.333990 17885 leader_election.cc:302] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8cb03b7c104cc4bf844d3e9d9f33f2, 809d6c5efa0349f684708986558bad8e; no voters:
I20260620 18:44:22.334264 18127 raft_consensus.cc:2781] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:22.334398 18127 raft_consensus.cc:482] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:22.334524 18127 raft_consensus.cc:3037] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:22.336812 18127 raft_consensus.cc:504] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.337353 18127 leader_election.cc:288] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [CANDIDATE]: Term 1 election: Requested vote from peers f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345), 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285)
I20260620 18:44:22.337793 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995cfb43dca94ab4a5afbdb9e9e82ef0" candidate_uuid: "809d6c5efa0349f684708986558bad8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5eb54b34c054558ac49176d4a270275"
I20260620 18:44:22.337883 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995cfb43dca94ab4a5afbdb9e9e82ef0" candidate_uuid: "809d6c5efa0349f684708986558bad8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2"
I20260620 18:44:22.337976 17678 raft_consensus.cc:3037] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:22.338102 17818 raft_consensus.cc:3037] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:22.340061 17678 raft_consensus.cc:2454] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809d6c5efa0349f684708986558bad8e in term 1.
I20260620 18:44:22.340221 17818 raft_consensus.cc:2454] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 809d6c5efa0349f684708986558bad8e in term 1.
I20260620 18:44:22.340556 17886 leader_election.cc:302] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 809d6c5efa0349f684708986558bad8e, f5eb54b34c054558ac49176d4a270275; no voters:
I20260620 18:44:22.340852 18127 raft_consensus.cc:2781] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:22.341274 18127 raft_consensus.cc:686] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 1 LEADER]: Becoming Leader. State: Replica: 809d6c5efa0349f684708986558bad8e, State: Running, Role: LEADER
I20260620 18:44:22.341591 18127 consensus_queue.cc:237] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } }
I20260620 18:44:22.344117 17517 catalog_manager.cc:5495] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 809d6c5efa0349f684708986558bad8e (127.15.102.195). New cstate: current_term: 1 leader_uuid: "809d6c5efa0349f684708986558bad8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "809d6c5efa0349f684708986558bad8e" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33445 } health_report { overall_health: HEALTHY } } }
I20260620 18:44:22.370726 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:22.365163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.15.102.195:33445)
0620 18:44:22.365721 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:22.365725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:22.365744 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:44:22.365799 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:22.365802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:22.365864 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:22.365957 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:22.366303 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:22.367556 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:22.368909 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:22.369371 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:22.369501 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:22.369969 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:22.370078 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:22.370145 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:22.370278 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 18:44:22.370339 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":336,"threads_started":1}
I20260620 18:44:22.379881 17818 raft_consensus.cc:1261] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Refusing update from remote peer 809d6c5efa0349f684708986558bad8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:22.380458 17678 raft_consensus.cc:1261] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Refusing update from remote peer 809d6c5efa0349f684708986558bad8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:22.380427 18138 consensus_queue.cc:1035] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8cb03b7c104cc4bf844d3e9d9f33f2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:22.381022 18127 consensus_queue.cc:1035] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eb54b34c054558ac49176d4a270275" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:44:22.399088 18092 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 18:44:23.977026 17517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:23.977301 17517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:23.981223 17517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9f4a036e6fe430fa4b25d34bf4faf5d: Sending DeleteTablet for 3 replicas of tablet 995cfb43dca94ab4a5afbdb9e9e82ef0
I20260620 18:44:23.982038 17654 tablet_service.cc:1514] Processing DeleteTablet for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:23 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:33292
I20260620 18:44:23.982132 17932 tablet_service.cc:1514] Processing DeleteTablet for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:23 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:38700
I20260620 18:44:23.982153 17793 tablet_service.cc:1514] Processing DeleteTablet for tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:23 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:54766
I20260620 18:44:23.982766 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17588
I20260620 18:44:23.986249 18166 tablet_replica.cc:330] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: stopping tablet replica
I20260620 18:44:23.986476 18166 raft_consensus.cc:2229] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:23.986828 18166 raft_consensus.cc:2258] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:23.987408 18167 tablet_replica.cc:330] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: stopping tablet replica
I20260620 18:44:23.987651 18167 raft_consensus.cc:2229] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:23.987885 18167 raft_consensus.cc:2258] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:23.988453 18166 ts_tablet_manager.cc:1905] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:23.989181 18167 ts_tablet_manager.cc:1905] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:23.994913 18165 tablet_replica.cc:330] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: stopping tablet replica
I20260620 18:44:23.995441 18166 ts_tablet_manager.cc:1918] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:44:23.995577 18166 log.cc:1196] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/995cfb43dca94ab4a5afbdb9e9e82ef0
I20260620 18:44:23.995630 18165 raft_consensus.cc:2229] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:23.995896 18165 raft_consensus.cc:2258] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:23.996037 18166 ts_tablet_manager.cc:1939] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 809d6c5efa0349f684708986558bad8e: Deleting consensus metadata
I20260620 18:44:23.996618 18167 ts_tablet_manager.cc:1918] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:44:23.996826 18167 log.cc:1196] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/995cfb43dca94ab4a5afbdb9e9e82ef0
I20260620 18:44:23.996897 18165 ts_tablet_manager.cc:1905] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:23.997161 17504 catalog_manager.cc:4841] TS 809d6c5efa0349f684708986558bad8e (127.15.102.195:33445): tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 (table test-table [id=13a26b9c819147239db03c7433f518b8]) successfully deleted
I20260620 18:44:23.997480 18167 ts_tablet_manager.cc:1939] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P 4c8cb03b7c104cc4bf844d3e9d9f33f2: Deleting consensus metadata
I20260620 18:44:23.998657 17503 catalog_manager.cc:4841] TS 4c8cb03b7c104cc4bf844d3e9d9f33f2 (127.15.102.194:45285): tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 (table test-table [id=13a26b9c819147239db03c7433f518b8]) successfully deleted
Jun 20 18:44:23 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:24.002653 18165 ts_tablet_manager.cc:1918] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 18:44:24.002817 18165 log.cc:1196] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/995cfb43dca94ab4a5afbdb9e9e82ef0
I20260620 18:44:24.003398 18165 ts_tablet_manager.cc:1939] T 995cfb43dca94ab4a5afbdb9e9e82ef0 P f5eb54b34c054558ac49176d4a270275: Deleting consensus metadata
I20260620 18:44:24.004498 17504 catalog_manager.cc:4841] TS f5eb54b34c054558ac49176d4a270275 (127.15.102.193:35345): tablet 995cfb43dca94ab4a5afbdb9e9e82ef0 (table test-table [id=13a26b9c819147239db03c7433f518b8]) successfully deleted
I20260620 18:44:24.007570 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:23.984280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.15.102.193:35345)
0620 18:44:23.985847 (+ 1567us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:23.985856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:23.987685 (+ 1829us) server_negotiation.cc:407] Connection header received
0620 18:44:23.987997 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:23.988006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:23.988142 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:23.988245 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:23.990484 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:23.992051 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:23.994626 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:23.995634 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:23.995761 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:24.001045 (+ 5284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:24.001067 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:24.001095 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:24.001147 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:24.005027 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:24.005664 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:24.005671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:24.005673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:24.005793 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:24.006165 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:24.006169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:24.006171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:24.006515 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:24.006722 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:24.007074 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 18:44:24.007304 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1389,"thread_start_us":1274,"threads_started":1}
W20260620 18:44:24.008201 17719 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46266
W20260620 18:44:24.008785 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17588: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:24.009011 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 17588
I20260620 18:44:24.031610 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17728
Jun 20 18:44:24 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:44:24.046212 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:24.032380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.15.102.194:45285)
0620 18:44:24.032876 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:24.032881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:24.032896 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:44:24.032950 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:24.032954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:24.033015 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:24.033103 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:24.034305 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:24.035026 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:24.036342 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:24.036735 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:24.036827 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:24.040916 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:24.040929 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:24.040950 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:24.040984 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:24.043771 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:24.044402 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:24.044409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:24.044412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:24.044565 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:24.044857 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:24.044860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:24.044862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:24.045194 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:24.045438 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:24.045660 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 18:44:24.045847 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
W20260620 18:44:24.046886 17858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49352
W20260620 18:44:24.047683 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17728: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:24.047947 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 17728
I20260620 18:44:24.070290 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17867
Jun 20 18:44:24 dist-test-slave-zfp8 krb5kdc[17472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:44:24.085239 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:24.071044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.15.102.195:33445)
0620 18:44:24.071593 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:24.071598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:24.071612 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:44:24.071663 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:24.071667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:24.071722 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:24.071840 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:24.073181 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:24.073903 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:24.075152 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:24.075652 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:24.075806 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:24.080038 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:24.080053 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:24.080070 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:24.080110 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:24.082688 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:24.083443 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:24.083450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:24.083452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:24.083569 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:24.083978 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:24.083981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:24.083983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:24.084307 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:24.084510 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:24.084785 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:44:24.084980 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":303,"threads_started":1}
W20260620 18:44:24.085909 17997 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39246
W20260620 18:44:24.086535 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17867: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:24.086678 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 17867
I20260620 18:44:24.108593 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 17485
I20260620 18:44:24.109762 17557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:24.225742 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 17485
2026-06-20T18:44:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8370 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:24 dist-test-slave-zfp8 krb5kdc[18175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:24 dist-test-slave-zfp8 krb5kdc[18175](info): set up 2 sockets
Jun 20 18:44:24 dist-test-slave-zfp8 krb5kdc[18175](info): commencing operation
krb5kdc: starting...
W20260620 18:44:25.319772 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:25 dist-test-slave-zfp8 krb5kdc[18175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:25.504822 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40201
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:25.824115 18188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:25.824409 18188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:25.824574 18188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:25.831887 18188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:25.832027 18188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:25.832115 18188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:25.832219 18188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:25.832295 18188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:25.842208 18188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:40201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:25.843591 18188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:25.844939 18188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:25.857053 18193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:25.857069 18194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:25.857092 18196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:25.860754 18188 server_base.cc:1034] running on GCE node
I20260620 18:44:25.861586 18188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:25.863075 18188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:25.864279 18188 hybrid_clock.cc:648] HybridClock initialized: now 1781981065864218 us; error 78 us; skew 500 ppm
I20260620 18:44:25.864670 18188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:25 dist-test-slave-zfp8 krb5kdc[18175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981065, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:25.867959 18188 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:25.869529 18188 webserver.cc:460] Webserver started at http://127.15.102.254:46231/ using document root <none> and password file <none>
I20260620 18:44:25.870250 18188 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:25.870393 18188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:25.870702 18188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:25.873746 18188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "c87c6c8e176943a3b15fe1680174b8d7"
format_stamp: "Formatted at 2026-06-20 18:44:25 on dist-test-slave-zfp8"
server_key: "392f7c411736382354570a3d95aac680"
server_key_iv: "5a9800bfb303fc4acfdbb5ffd528e6b2"
server_key_version: "encryptionkey@0"
I20260620 18:44:25.874506 18188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "c87c6c8e176943a3b15fe1680174b8d7"
format_stamp: "Formatted at 2026-06-20 18:44:25 on dist-test-slave-zfp8"
server_key: "392f7c411736382354570a3d95aac680"
server_key_iv: "5a9800bfb303fc4acfdbb5ffd528e6b2"
server_key_version: "encryptionkey@0"
I20260620 18:44:25.879861 18188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:44:25.883378 18203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:25.884652 18188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:25.884828 18188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "c87c6c8e176943a3b15fe1680174b8d7"
format_stamp: "Formatted at 2026-06-20 18:44:25 on dist-test-slave-zfp8"
server_key: "392f7c411736382354570a3d95aac680"
server_key_iv: "5a9800bfb303fc4acfdbb5ffd528e6b2"
server_key_version: "encryptionkey@0"
I20260620 18:44:25.885032 18188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:25.928045 18188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:25.932214 18188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:25.932511 18188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:25.951037 18188 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:40201
I20260620 18:44:25.951061 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:40201 every 8 connection(s)
I20260620 18:44:25.952540 18188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:25.954061 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 18188
I20260620 18:44:25.954294 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:25.955039 15771 external_mini_cluster.cc:1428] Setting key 1305566b3d1c12097e7d2017bf80ecaa
I20260620 18:44:25.957409 18255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:44:25 dist-test-slave-zfp8 krb5kdc[18175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:25.968986 18255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7: Bootstrap starting.
I20260620 18:44:25.972379 18255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:25.973444 18255 log.cc:824] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:25.976387 18255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7: No bootstrap required, opened a new log
I20260620 18:44:25.977283 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:25.957182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.15.102.254:40201)
0620 18:44:25.957946 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:25.957958 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:25.957999 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:44:25.959718 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:25.959739 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:25.960120 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:25.960446 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:25.962148 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:25.963577 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:25.965025 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:25.965673 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:25.965878 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:25.970107 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:25.970115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:25.970128 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:25.970171 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:25.973268 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:25.974007 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:25.974015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:25.974021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:25.974253 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:25.974628 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:25.974634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:25.974637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:25.975128 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:25.975449 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:25.975860 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 18:44:25.976268 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":357,"threads_started":1}
I20260620 18:44:25.983831 18255 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c6c8e176943a3b15fe1680174b8d7" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40201 } }
I20260620 18:44:25.984210 18255 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:25.984318 18255 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87c6c8e176943a3b15fe1680174b8d7, State: Initialized, Role: FOLLOWER
I20260620 18:44:25.984983 18255 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c6c8e176943a3b15fe1680174b8d7" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40201 } }
I20260620 18:44:25.985208 18255 raft_consensus.cc:388] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:25.985365 18255 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:25.985539 18255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:25.987692 18255 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c6c8e176943a3b15fe1680174b8d7" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40201 } }
I20260620 18:44:25.988152 18255 leader_election.cc:302] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c87c6c8e176943a3b15fe1680174b8d7; no voters:
I20260620 18:44:25.988644 18255 leader_election.cc:288] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:25.988886 18260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:25.989804 18260 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [term 1 LEADER]: Becoming Leader. State: Replica: c87c6c8e176943a3b15fe1680174b8d7, State: Running, Role: LEADER
I20260620 18:44:25.990331 18260 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c6c8e176943a3b15fe1680174b8d7" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40201 } }
I20260620 18:44:25.990749 18255 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:25.993136 18262 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87c6c8e176943a3b15fe1680174b8d7. Latest consensus state: current_term: 1 leader_uuid: "c87c6c8e176943a3b15fe1680174b8d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c6c8e176943a3b15fe1680174b8d7" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40201 } } }
I20260620 18:44:25.993091 18261 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c87c6c8e176943a3b15fe1680174b8d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c6c8e176943a3b15fe1680174b8d7" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40201 } } }
I20260620 18:44:25.993543 18261 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:25.995054 18267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:25.995036 18262 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:25.999775 18267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:26.008167 18267 catalog_manager.cc:1347] Generated new cluster ID: 971d3977d8a64dda9853a8419dc4ca5e
I20260620 18:44:26.008296 18267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:26.020890 18267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:26.022253 18267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:26.031322 18267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87c6c8e176943a3b15fe1680174b8d7: Generated new TSK 0
I20260620 18:44:26.032184 18267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:26.080681 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40201
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:26.408511 18281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:26.408792 18281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:26.408951 18281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:26.416115 18281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:26.416229 18281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:26.416395 18281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:26.426187 18281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:26.427541 18281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:26.428879 18281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:26.442185 18286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:26.442167 18287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:26.442684 18289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:26.442960 18281 server_base.cc:1034] running on GCE node
I20260620 18:44:26.443629 18281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:26.444618 18281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:26.445853 18281 hybrid_clock.cc:648] HybridClock initialized: now 1781981066445820 us; error 48 us; skew 500 ppm
I20260620 18:44:26.446184 18281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:26 dist-test-slave-zfp8 krb5kdc[18175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981066, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:26.449281 18281 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:26.450843 18281 webserver.cc:460] Webserver started at http://127.15.102.193:43173/ using document root <none> and password file <none>
I20260620 18:44:26.451805 18281 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:26.451992 18281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:26.452381 18281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:26.456599 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "02377a6b618c435b87d5bfbcd14a5631"
format_stamp: "Formatted at 2026-06-20 18:44:26 on dist-test-slave-zfp8"
server_key: "bc6a9fa91dc37322c9fe82f7a84ccd30"
server_key_iv: "adb7a578d3dd0d74b5a54d4b70a478a2"
server_key_version: "encryptionkey@0"
I20260620 18:44:26.457783 18281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "02377a6b618c435b87d5bfbcd14a5631"
format_stamp: "Formatted at 2026-06-20 18:44:26 on dist-test-slave-zfp8"
server_key: "bc6a9fa91dc37322c9fe82f7a84ccd30"
server_key_iv: "adb7a578d3dd0d74b5a54d4b70a478a2"
server_key_version: "encryptionkey@0"
I20260620 18:44:26.463109 18281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:44:26.466584 18296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:26.467921 18281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:26.468045 18281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "02377a6b618c435b87d5bfbcd14a5631"
format_stamp: "Formatted at 2026-06-20 18:44:26 on dist-test-slave-zfp8"
server_key: "bc6a9fa91dc37322c9fe82f7a84ccd30"
server_key_iv: "adb7a578d3dd0d74b5a54d4b70a478a2"
server_key_version: "encryptionkey@0"
I20260620 18:44:26.468210 18281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:26.495631 18281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:26.499687 18281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:26.499917 18281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:26.500828 18281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:26.502055 18281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:26.502136 18281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:26.502218 18281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:26.502259 18281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:26.535820 18281 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:46229
I20260620 18:44:26.535866 18408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:46229 every 8 connection(s)
I20260620 18:44:26.537135 18281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:26.541493 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 18281
I20260620 18:44:26.541728 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:26.542517 15771 external_mini_cluster.cc:1428] Setting key 9640b58337e95908e3d4a8dd8266e71a
Jun 20 18:44:26 dist-test-slave-zfp8 krb5kdc[18175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981066, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:26.560057 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:26.539769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:33101 (local address 127.15.102.254:40201)
0620 18:44:26.540248 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:26.540253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:26.541179 (+ 926us) server_negotiation.cc:407] Connection header received
0620 18:44:26.543697 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:26.543702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:26.543765 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:26.543856 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:26.545832 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:26.546532 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:26.549057 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:26.549447 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:26.549545 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:26.553126 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:26.553134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:26.553142 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:26.553187 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:26.556790 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:26.557531 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:26.557540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:26.557546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:26.557706 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:26.558114 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:26.558122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:26.558128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:26.558529 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:26.558728 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:26.559397 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 18:44:26.559637 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":263,"threads_started":1}
I20260620 18:44:26.560758 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:26.540248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40201 (local address 127.15.102.193:33101)
0620 18:44:26.540906 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:26.541036 (+ 130us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:26.543414 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:26.544132 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:26.544144 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:26.544815 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:26.545652 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:26.545680 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:26.546697 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:26.546705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:26.548913 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:26.548931 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:26.549683 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:26.549692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:26.550021 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:26.550736 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:26.550759 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:26.552899 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:26.556929 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:26.556936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:26.556949 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:26.557353 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:26.557817 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:26.557820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:26.557822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:26.557989 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:26.558663 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:26.558668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:26.558980 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 18:44:26.559198 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 18:44:26.559641 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":253,"threads_started":1}
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:44:26.562413 18409 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40201
I20260620 18:44:26.562726 18409 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:26.563500 18409 heartbeater.cc:511] Master 127.15.102.254:40201 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
I20260620 18:44:26.565670 18220 ts_manager.cc:194] Registered new tserver with Master: 02377a6b618c435b87d5bfbcd14a5631 (127.15.102.193:46229)
I20260620 18:44:26.567081 18220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:33101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:26.586341 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40201
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:26.903007 18415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:26.903357 18415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:26.903530 18415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:26.910518 18415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:26.910669 18415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:26.910861 18415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:26.920229 18415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:26.921509 18415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:26.922834 18415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:26.935784 18421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:26.935863 18420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:26.936057 18423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:26.936817 18415 server_base.cc:1034] running on GCE node
I20260620 18:44:26.937441 18415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:26.938385 18415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:26.939569 18415 hybrid_clock.cc:648] HybridClock initialized: now 1781981066939505 us; error 66 us; skew 500 ppm
I20260620 18:44:26.939877 18415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:26 dist-test-slave-zfp8 krb5kdc[18175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981066, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:26.942914 18415 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:26.944451 18415 webserver.cc:460] Webserver started at http://127.15.102.194:34839/ using document root <none> and password file <none>
I20260620 18:44:26.945132 18415 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:26.945271 18415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:26.945569 18415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:26.948436 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "1552d7e0b8f546dfb0a2d6c07e7eea5d"
format_stamp: "Formatted at 2026-06-20 18:44:26 on dist-test-slave-zfp8"
server_key: "ca2cccddcf345bca345415529eac4878"
server_key_iv: "1265953eef595bcd3df71080a360b031"
server_key_version: "encryptionkey@0"
I20260620 18:44:26.949201 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "1552d7e0b8f546dfb0a2d6c07e7eea5d"
format_stamp: "Formatted at 2026-06-20 18:44:26 on dist-test-slave-zfp8"
server_key: "ca2cccddcf345bca345415529eac4878"
server_key_iv: "1265953eef595bcd3df71080a360b031"
server_key_version: "encryptionkey@0"
I20260620 18:44:26.954329 18415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:44:26.957790 18430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:26.959137 18415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:26.959334 18415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "1552d7e0b8f546dfb0a2d6c07e7eea5d"
format_stamp: "Formatted at 2026-06-20 18:44:26 on dist-test-slave-zfp8"
server_key: "ca2cccddcf345bca345415529eac4878"
server_key_iv: "1265953eef595bcd3df71080a360b031"
server_key_version: "encryptionkey@0"
I20260620 18:44:26.959533 18415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:26.980335 18415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:26.984254 18415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:26.984542 18415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:26.985481 18415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:26.986778 18415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:26.986847 18415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:26.987020 18415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:26.987087 18415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:27.018536 18415 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:40383
I20260620 18:44:27.018572 18542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:40383 every 8 connection(s)
I20260620 18:44:27.019901 18415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:27.023676 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 18415
I20260620 18:44:27.023876 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:27.024438 15771 external_mini_cluster.cc:1428] Setting key e006e6f7e51e71e01e7e3f78b4866252
Jun 20 18:44:27 dist-test-slave-zfp8 krb5kdc[18175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981066, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:27.042099 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:27.022491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:59415 (local address 127.15.102.254:40201)
0620 18:44:27.022713 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:27.022720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:27.024202 (+ 1482us) server_negotiation.cc:407] Connection header received
0620 18:44:27.026396 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:27.026401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:27.026479 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:27.026591 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:27.028148 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:27.028845 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:27.030374 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:27.031014 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:27.031153 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:27.034714 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:27.034723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:27.034731 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:27.034775 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:27.038497 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:27.039409 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:27.039416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:27.039423 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:27.039587 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:27.040050 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:27.040058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:27.040063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:27.040456 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:27.040619 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:27.041455 (+ 836us) server_negotiation.cc:299] Negotiation successful
0620 18:44:27.041695 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
I20260620 18:44:27.043162 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:27.023099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40201 (local address 127.15.102.194:59415)
0620 18:44:27.024018 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:27.024078 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:27.026108 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:27.026837 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:27.026846 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:27.027411 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:27.027980 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:27.027997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:27.029000 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:27.029005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:27.030230 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:27.030242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:27.031281 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:27.031291 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:27.031625 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:27.032363 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:27.032385 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:27.034484 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:27.038679 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:27.038691 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:27.038723 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:27.039203 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:27.039727 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:27.039730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:27.039733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:27.039918 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:27.040629 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:27.040636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:27.041035 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 18:44:27.041307 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 18:44:27.041742 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":368,"threads_started":1}
I20260620 18:44:27.044854 18543 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
I20260620 18:44:27.045192 18543 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:27.045902 18543 heartbeater.cc:511] Master 127.15.102.254:40201 requested a full tablet report, sending...
I20260620 18:44:27.047436 18220 ts_manager.cc:194] Registered new tserver with Master: 1552d7e0b8f546dfb0a2d6c07e7eea5d (127.15.102.194:40383)
I20260620 18:44:27.048233 18220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:59415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:27.068367 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40201
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:27.389262 18548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:27.389540 18548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:27.389698 18548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:27.396997 18548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:27.397109 18548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:27.397222 18548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:27.406819 18548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:27.408162 18548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:27.409402 18548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:27.422319 18554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:27.422317 18556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:27.422554 18553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:27.423221 18548 server_base.cc:1034] running on GCE node
I20260620 18:44:27.423892 18548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:27.424885 18548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:27.426079 18548 hybrid_clock.cc:648] HybridClock initialized: now 1781981067426025 us; error 57 us; skew 500 ppm
I20260620 18:44:27.426385 18548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:27 dist-test-slave-zfp8 krb5kdc[18175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981067, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:27.429474 18548 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:27.430884 18548 webserver.cc:460] Webserver started at http://127.15.102.195:34693/ using document root <none> and password file <none>
I20260620 18:44:27.431684 18548 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:27.431807 18548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:27.432132 18548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:27.435009 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "3c63c07da0f94e8fa7ec05082f081b68"
format_stamp: "Formatted at 2026-06-20 18:44:27 on dist-test-slave-zfp8"
server_key: "51128d71a97ee7fad3c85f44635079fd"
server_key_iv: "6598a65e660d9b0371acc800a4127593"
server_key_version: "encryptionkey@0"
I20260620 18:44:27.435866 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "3c63c07da0f94e8fa7ec05082f081b68"
format_stamp: "Formatted at 2026-06-20 18:44:27 on dist-test-slave-zfp8"
server_key: "51128d71a97ee7fad3c85f44635079fd"
server_key_iv: "6598a65e660d9b0371acc800a4127593"
server_key_version: "encryptionkey@0"
I20260620 18:44:27.440845 18548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:44:27.444337 18563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:27.445760 18548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:44:27.445919 18548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "3c63c07da0f94e8fa7ec05082f081b68"
format_stamp: "Formatted at 2026-06-20 18:44:27 on dist-test-slave-zfp8"
server_key: "51128d71a97ee7fad3c85f44635079fd"
server_key_iv: "6598a65e660d9b0371acc800a4127593"
server_key_version: "encryptionkey@0"
I20260620 18:44:27.446126 18548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:27.471103 18548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:27.474984 18548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:27.475337 18548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:27.476270 18548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:27.477530 18548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:27.477598 18548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:27.477766 18548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:27.477816 18548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:27.510681 18548 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:35241
I20260620 18:44:27.510730 18675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:35241 every 8 connection(s)
I20260620 18:44:27.511998 18548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:27.516561 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 18548
I20260620 18:44:27.516753 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:27.517328 15771 external_mini_cluster.cc:1428] Setting key 7b38a75b8354cdd0f9e2756e497a53d7
Jun 20 18:44:27 dist-test-slave-zfp8 krb5kdc[18175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981067, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:27.531889 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:27.514483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:60903 (local address 127.15.102.254:40201)
0620 18:44:27.514684 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:27.514690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:27.516026 (+ 1336us) server_negotiation.cc:407] Connection header received
0620 18:44:27.518576 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:27.518581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:27.518681 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:27.518775 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:27.520580 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:27.521301 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:27.522744 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:27.523171 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:27.523327 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:27.526520 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:27.526524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:27.526527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:27.526559 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:27.528896 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:27.529559 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:27.529562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:27.529564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:27.529681 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:27.530025 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:27.530027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:27.530029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:27.530282 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:27.530383 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:27.531346 (+ 963us) server_negotiation.cc:299] Negotiation successful
0620 18:44:27.531592 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:44:27.533059 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:27.515056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40201 (local address 127.15.102.195:60903)
0620 18:44:27.515866 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:27.515909 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:27.518283 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:27.519053 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:27.519067 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:27.519695 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:27.520435 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:27.520454 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:27.521413 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:27.521417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:27.522622 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:27.522633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:27.523412 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:27.523417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:27.523609 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:27.524278 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:27.524300 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:27.526333 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:27.529015 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:27.529021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:27.529034 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:27.529447 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:27.529768 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:27.529771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:27.529774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:27.529932 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:27.530417 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:27.530424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:27.530825 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 18:44:27.531146 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 18:44:27.531704 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":362,"threads_started":1}
I20260620 18:44:27.534318 18676 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40201
I20260620 18:44:27.534621 18676 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:27.535377 18676 heartbeater.cc:511] Master 127.15.102.254:40201 requested a full tablet report, sending...
I20260620 18:44:27.536733 18220 ts_manager.cc:194] Registered new tserver with Master: 3c63c07da0f94e8fa7ec05082f081b68 (127.15.102.195:35241)
I20260620 18:44:27.537493 18220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:60903
I20260620 18:44:27.544960 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:44:27.570092 18409 heartbeater.cc:503] Master 127.15.102.254:40201 was elected leader, sending a full tablet report...
W20260620 18:44:27.574851 18411 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:44:27.572422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.15.102.254:40201)
0620 18:44:27.572618 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:27.572627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:27.573096 (+ 469us) server_negotiation.cc:407] Connection header received
0620 18:44:27.573240 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:27.573243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:27.573299 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:27.573380 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:27.574594 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42508: BlockingRecv error: recv got EOF from 127.0.0.1:42508 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:44:27.578377 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18281
W20260620 18:44:27.581105 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18281: Not authorized: Client connection negotiation failed: client connection to 127.15.102.193:46229: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:44:27.581286 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 18281
I20260620 18:44:27.597360 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18415
W20260620 18:44:27.599953 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18415: Not authorized: Client connection negotiation failed: client connection to 127.15.102.194:40383: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:44:27.600154 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 18415
I20260620 18:44:27.615212 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18548
W20260620 18:44:27.618286 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18548: Not authorized: Client connection negotiation failed: client connection to 127.15.102.195:35241: server requires authentication, but client does not have Kerberos credentials available
I20260620 18:44:27.618441 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 18548
I20260620 18:44:27.634362 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18188
I20260620 18:44:27.635581 18250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:27.735718 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 18188
2026-06-20T18:44:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3506 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:27 dist-test-slave-zfp8 krb5kdc[18693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:27 dist-test-slave-zfp8 krb5kdc[18693](info): set up 2 sockets
Jun 20 18:44:27 dist-test-slave-zfp8 krb5kdc[18693](info): commencing operation
krb5kdc: starting...
W20260620 18:44:28.815706 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:28 dist-test-slave-zfp8 krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:28.999527 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:42449
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:29.317893 18706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:29.318244 18706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:29.318395 18706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:29.325783 18706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:29.325935 18706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:29.326042 18706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:29.326150 18706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:29.326231 18706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:29.336292 18706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:42449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:29.337653 18706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:29.338995 18706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:29.350559 18711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:29.350616 18712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:29.350513 18714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:29.351753 18706 server_base.cc:1034] running on GCE node
I20260620 18:44:29.352569 18706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:29.353996 18706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:29.355192 18706 hybrid_clock.cc:648] HybridClock initialized: now 1781981069355151 us; error 33 us; skew 500 ppm
I20260620 18:44:29.355574 18706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:29 dist-test-slave-zfp8 krb5kdc[18693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981069, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:29.358567 18706 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:29.360188 18706 webserver.cc:460] Webserver started at http://127.15.102.254:46235/ using document root <none> and password file <none>
I20260620 18:44:29.360872 18706 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:29.361013 18706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:29.361330 18706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:29.364349 18706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "734bc51ac94646ae953c9b0ebce2daa9"
format_stamp: "Formatted at 2026-06-20 18:44:29 on dist-test-slave-zfp8"
server_key: "1ef50eccf7c28ff1b680258b939fedd5"
server_key_iv: "db0181ecf96cb801a3d05b8bde329232"
server_key_version: "encryptionkey@0"
I20260620 18:44:29.365132 18706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "734bc51ac94646ae953c9b0ebce2daa9"
format_stamp: "Formatted at 2026-06-20 18:44:29 on dist-test-slave-zfp8"
server_key: "1ef50eccf7c28ff1b680258b939fedd5"
server_key_iv: "db0181ecf96cb801a3d05b8bde329232"
server_key_version: "encryptionkey@0"
I20260620 18:44:29.370206 18706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:44:29.373919 18721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:29.375324 18706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:44:29.375502 18706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "734bc51ac94646ae953c9b0ebce2daa9"
format_stamp: "Formatted at 2026-06-20 18:44:29 on dist-test-slave-zfp8"
server_key: "1ef50eccf7c28ff1b680258b939fedd5"
server_key_iv: "db0181ecf96cb801a3d05b8bde329232"
server_key_version: "encryptionkey@0"
I20260620 18:44:29.375710 18706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:29.403550 18706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:29.407686 18706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:29.407991 18706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:29.427261 18706 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:42449
I20260620 18:44:29.427294 18772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:42449 every 8 connection(s)
I20260620 18:44:29.428653 18706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:29.432538 18773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:29.437578 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 18706
I20260620 18:44:29.437825 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:29.438593 15771 external_mini_cluster.cc:1428] Setting key 34df24e6dde8a5db9caa0fa1b9b5c7ff
I20260620 18:44:29.444176 18773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9: Bootstrap starting.
I20260620 18:44:29.447166 18773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:29.448150 18773 log.cc:824] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:29.451123 18773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9: No bootstrap required, opened a new log
Jun 20 18:44:29 dist-test-slave-zfp8 krb5kdc[18693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:29.459321 18773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734bc51ac94646ae953c9b0ebce2daa9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42449 } }
I20260620 18:44:29.459717 18773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:29.459834 18773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734bc51ac94646ae953c9b0ebce2daa9, State: Initialized, Role: FOLLOWER
I20260620 18:44:29.460565 18773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734bc51ac94646ae953c9b0ebce2daa9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42449 } }
I20260620 18:44:29.460852 18773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:29.461016 18773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:29.461199 18773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:29.462109 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:29.440669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.15.102.254:42449)
0620 18:44:29.441493 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:29.441508 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:29.441552 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:44:29.443376 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:29.443400 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:29.443787 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:29.444193 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:29.445668 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:29.446964 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:29.448306 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:29.449336 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:29.449514 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:29.453855 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:29.453864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:29.453880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:29.453934 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:29.457431 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:29.458232 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:29.458253 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:29.458260 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:29.458440 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:29.458890 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:29.458898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:29.458901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:29.459567 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:29.459951 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:29.460452 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 18:44:29.460896 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":391,"threads_started":1}
I20260620 18:44:29.464066 18773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734bc51ac94646ae953c9b0ebce2daa9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42449 } }
I20260620 18:44:29.464617 18773 leader_election.cc:302] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 734bc51ac94646ae953c9b0ebce2daa9; no voters:
I20260620 18:44:29.465236 18773 leader_election.cc:288] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:29.465363 18778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:29.466316 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:44:29.466507 18778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [term 1 LEADER]: Becoming Leader. State: Replica: 734bc51ac94646ae953c9b0ebce2daa9, State: Running, Role: LEADER
I20260620 18:44:29.466975 18778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734bc51ac94646ae953c9b0ebce2daa9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42449 } }
I20260620 18:44:29.467435 18773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:29.471731 18780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 734bc51ac94646ae953c9b0ebce2daa9. Latest consensus state: current_term: 1 leader_uuid: "734bc51ac94646ae953c9b0ebce2daa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734bc51ac94646ae953c9b0ebce2daa9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42449 } } }
I20260620 18:44:29.471962 18779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "734bc51ac94646ae953c9b0ebce2daa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734bc51ac94646ae953c9b0ebce2daa9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42449 } } }
I20260620 18:44:29.472164 18780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:29.472234 18779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:29.473403 18787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:29.478389 18787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:29.494508 18787 catalog_manager.cc:1347] Generated new cluster ID: d52aa7c9ee004e64ac44bc6a1657f8f0
I20260620 18:44:29.494742 18787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:29.508144 18787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:29.510620 18787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:29.521070 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:29.513237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.15.102.254:42449)
0620 18:44:29.513427 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:29.513435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:29.514296 (+ 861us) server_negotiation.cc:407] Connection header received
0620 18:44:29.514515 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:29.514520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:29.514595 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:29.514703 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:29.515756 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:29.516480 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:29.517776 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:29.518305 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:29.518445 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:29.519609 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:29.519619 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:29.519623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:29.520253 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:29.520316 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:29.520347 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:29.520489 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:29.520612 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 18:44:29.520711 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:44:29.524044 18787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 734bc51ac94646ae953c9b0ebce2daa9: Generated new TSK 0
I20260620 18:44:29.525156 18787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:44:29.533193 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18706
I20260620 18:44:29.534746 18768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:29.622166 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 18706
2026-06-20T18:44:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1881 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:29 dist-test-slave-zfp8 krb5kdc[18807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:29 dist-test-slave-zfp8 krb5kdc[18807](info): set up 2 sockets
Jun 20 18:44:29 dist-test-slave-zfp8 krb5kdc[18807](info): commencing operation
krb5kdc: starting...
W20260620 18:44:30.699692 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:30 dist-test-slave-zfp8 krb5kdc[18807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:30.886186 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:38453
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:38453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:31.200896 18820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:31.201184 18820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:31.201288 18820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:31.208364 18820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:31.208472 18820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:31.208520 18820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:31.208577 18820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:31.208613 18820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:31.218214 18820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:38453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:31.219532 18820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:31.220772 18820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:31.232620 18826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:31.232697 18828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:31.232688 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:31.233615 18820 server_base.cc:1034] running on GCE node
I20260620 18:44:31.234333 18820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:31.235786 18820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:31.236992 18820 hybrid_clock.cc:648] HybridClock initialized: now 1781981071236950 us; error 34 us; skew 500 ppm
I20260620 18:44:31.237352 18820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:31 dist-test-slave-zfp8 krb5kdc[18807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981071, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:31.240401 18820 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:31.242254 18820 webserver.cc:460] Webserver started at http://127.15.102.254:37233/ using document root <none> and password file <none>
I20260620 18:44:31.242914 18820 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:31.243010 18820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:31.243335 18820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:31.246229 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "241f35982ba24bae95eb4eeacf8d6dc5"
format_stamp: "Formatted at 2026-06-20 18:44:31 on dist-test-slave-zfp8"
server_key: "851cc844afa1ab0140c9bbaac684904d"
server_key_iv: "3981de239bef3b63cf5243cf1e1a725b"
server_key_version: "encryptionkey@0"
I20260620 18:44:31.246902 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "241f35982ba24bae95eb4eeacf8d6dc5"
format_stamp: "Formatted at 2026-06-20 18:44:31 on dist-test-slave-zfp8"
server_key: "851cc844afa1ab0140c9bbaac684904d"
server_key_iv: "3981de239bef3b63cf5243cf1e1a725b"
server_key_version: "encryptionkey@0"
I20260620 18:44:31.251881 18820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:44:31.256140 18835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:31.257617 18820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260620 18:44:31.257802 18820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "241f35982ba24bae95eb4eeacf8d6dc5"
format_stamp: "Formatted at 2026-06-20 18:44:31 on dist-test-slave-zfp8"
server_key: "851cc844afa1ab0140c9bbaac684904d"
server_key_iv: "3981de239bef3b63cf5243cf1e1a725b"
server_key_version: "encryptionkey@0"
I20260620 18:44:31.257998 18820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:31.274165 18820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:31.278311 18820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:31.278592 18820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:31.297147 18820 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:38453
I20260620 18:44:31.297314 18886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:38453 every 8 connection(s)
I20260620 18:44:31.298539 18820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:31.302189 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:31.303725 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 18820
I20260620 18:44:31.303944 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:31.304569 15771 external_mini_cluster.cc:1428] Setting key af36e26e858b812b6ae39180ecaeba67
I20260620 18:44:31.314227 18887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5: Bootstrap starting.
I20260620 18:44:31.318216 18887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5: Neither blocks nor log segments found. Creating new log.
Jun 20 18:44:31 dist-test-slave-zfp8 krb5kdc[18807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:31.319453 18887 log.cc:824] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:31.323103 18887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5: No bootstrap required, opened a new log
I20260620 18:44:31.328809 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:31.306481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.15.102.254:38453)
0620 18:44:31.307310 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:31.307322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:31.307371 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:44:31.309007 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:31.309029 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:31.309439 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:31.309755 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:31.311544 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:31.312856 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:31.314558 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:31.315317 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:31.315465 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:31.320157 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:31.320168 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:31.320187 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:31.320227 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:31.324004 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:31.324723 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:31.324733 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:31.324743 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:31.324947 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:31.325401 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:31.325406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:31.325409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:31.326155 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:31.326494 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:31.326828 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:44:31.327340 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":368,"threads_started":1}
I20260620 18:44:31.333201 18887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241f35982ba24bae95eb4eeacf8d6dc5" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38453 } }
I20260620 18:44:31.333547 18887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:31.333621 18887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241f35982ba24bae95eb4eeacf8d6dc5, State: Initialized, Role: FOLLOWER
I20260620 18:44:31.334139 18887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241f35982ba24bae95eb4eeacf8d6dc5" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38453 } }
I20260620 18:44:31.334332 18887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:31.334429 18887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:31.334544 18887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:31.336815 18887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241f35982ba24bae95eb4eeacf8d6dc5" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38453 } }
I20260620 18:44:31.337430 18887 leader_election.cc:302] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 241f35982ba24bae95eb4eeacf8d6dc5; no voters:
I20260620 18:44:31.338027 18887 leader_election.cc:288] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:31.338179 18892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:31.338864 18892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [term 1 LEADER]: Becoming Leader. State: Replica: 241f35982ba24bae95eb4eeacf8d6dc5, State: Running, Role: LEADER
I20260620 18:44:31.339475 18892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241f35982ba24bae95eb4eeacf8d6dc5" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38453 } }
I20260620 18:44:31.340178 18887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:31.342245 18893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "241f35982ba24bae95eb4eeacf8d6dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241f35982ba24bae95eb4eeacf8d6dc5" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38453 } } }
I20260620 18:44:31.342589 18893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:31.342425 18894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 241f35982ba24bae95eb4eeacf8d6dc5. Latest consensus state: current_term: 1 leader_uuid: "241f35982ba24bae95eb4eeacf8d6dc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "241f35982ba24bae95eb4eeacf8d6dc5" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38453 } } }
I20260620 18:44:31.342816 18894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:31.344419 18897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:31.349591 18897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:31.359265 18897 catalog_manager.cc:1347] Generated new cluster ID: 7caf8e2336b34278b021de267fda4204
I20260620 18:44:31.359422 18897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:31.368503 18897 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:31.370389 18897 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:31.386641 18897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 241f35982ba24bae95eb4eeacf8d6dc5: Generated new TSK 0
I20260620 18:44:31.387801 18897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:31.430297 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38453
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:31.756796 18913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:31.757066 18913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:31.757253 18913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:31.764709 18913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:31.764825 18913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:31.765023 18913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:31.774827 18913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:31.776254 18913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:31.777710 18913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:31.790697 18918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:31.790663 18919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:31.790733 18921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:31.791818 18913 server_base.cc:1034] running on GCE node
I20260620 18:44:31.792426 18913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:31.793362 18913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:31.794584 18913 hybrid_clock.cc:648] HybridClock initialized: now 1781981071794529 us; error 56 us; skew 500 ppm
I20260620 18:44:31.794901 18913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:31 dist-test-slave-zfp8 krb5kdc[18807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981071, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:31.798070 18913 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:31.799445 18913 webserver.cc:460] Webserver started at http://127.15.102.193:34361/ using document root <none> and password file <none>
I20260620 18:44:31.800102 18913 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:31.800202 18913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:31.800449 18913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:31.803186 18913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "703d7ce12ab7401e83249c35479b4034"
format_stamp: "Formatted at 2026-06-20 18:44:31 on dist-test-slave-zfp8"
server_key: "e3ed15fe52c4dee6e2c37e9965492f06"
server_key_iv: "a7d470396c16f9413272f4fe4d113e83"
server_key_version: "encryptionkey@0"
I20260620 18:44:31.803963 18913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "703d7ce12ab7401e83249c35479b4034"
format_stamp: "Formatted at 2026-06-20 18:44:31 on dist-test-slave-zfp8"
server_key: "e3ed15fe52c4dee6e2c37e9965492f06"
server_key_iv: "a7d470396c16f9413272f4fe4d113e83"
server_key_version: "encryptionkey@0"
I20260620 18:44:31.808864 18913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:44:31.812433 18928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:31.813768 18913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:44:31.813932 18913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "703d7ce12ab7401e83249c35479b4034"
format_stamp: "Formatted at 2026-06-20 18:44:31 on dist-test-slave-zfp8"
server_key: "e3ed15fe52c4dee6e2c37e9965492f06"
server_key_iv: "a7d470396c16f9413272f4fe4d113e83"
server_key_version: "encryptionkey@0"
I20260620 18:44:31.814148 18913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:31.827597 18913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:31.832105 18913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:31.832350 18913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:31.833214 18913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:31.834470 18913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:31.834540 18913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:31.834623 18913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:31.834671 18913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:31.867161 18913 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:45179
I20260620 18:44:31.867204 19040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:45179 every 8 connection(s)
I20260620 18:44:31.868549 18913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:31.878949 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 18913
I20260620 18:44:31.879154 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:31.879833 15771 external_mini_cluster.cc:1428] Setting key c9c73fd478eef4ccc8e954b34f63052c
Jun 20 18:44:31 dist-test-slave-zfp8 krb5kdc[18807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981071, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:31.888968 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:31.871006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:47839 (local address 127.15.102.254:38453)
0620 18:44:31.871476 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:31.871480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:31.872379 (+ 899us) server_negotiation.cc:407] Connection header received
0620 18:44:31.874112 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:31.874118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:31.874221 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:31.874338 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:31.875855 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:31.876565 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:31.878217 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:31.878632 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:31.878763 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:31.882577 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:31.882583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:31.882587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:31.882620 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:31.885220 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:31.886121 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:31.886125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:31.886128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:31.886291 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:31.886945 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:31.886952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:31.886956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:31.887398 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:31.887557 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:31.888224 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 18:44:31.888488 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":275,"threads_started":1}
I20260620 18:44:31.889689 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:31.871535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:38453 (local address 127.15.102.193:47839)
0620 18:44:31.872220 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:31.872259 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:31.873850 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:31.874568 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:31.874577 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:31.875072 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:31.875692 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:31.875709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:31.876731 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:31.876735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:31.878070 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:31.878081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:31.878877 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:31.878884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:31.879154 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:31.879943 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:31.879979 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:31.882329 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:31.885386 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:31.885396 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:31.885417 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:31.885978 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:31.886451 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:31.886454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:31.886457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:31.886636 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:31.887517 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:31.887523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:31.887853 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 18:44:31.888076 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 18:44:31.888474 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":269,"threads_started":1}
I20260620 18:44:31.891222 19041 heartbeater.cc:347] Connected to a master server at 127.15.102.254:38453
I20260620 18:44:31.891574 19041 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:31.892309 19041 heartbeater.cc:511] Master 127.15.102.254:38453 requested a full tablet report, sending...
I20260620 18:44:31.894408 18852 ts_manager.cc:194] Registered new tserver with Master: 703d7ce12ab7401e83249c35479b4034 (127.15.102.193:45179)
I20260620 18:44:31.895884 18852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:47839
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:31.920487 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38453
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:32.240746 19047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:32.241040 19047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:32.241231 19047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:32.248667 19047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:32.248777 19047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:32.248989 19047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:32.260281 19047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:32.261759 19047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:32.263185 19047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:32.276623 19055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:32.276561 19053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:32.277112 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:32.277230 19047 server_base.cc:1034] running on GCE node
I20260620 18:44:32.277920 19047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:32.278882 19047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:32.280107 19047 hybrid_clock.cc:648] HybridClock initialized: now 1781981072280060 us; error 59 us; skew 500 ppm
I20260620 18:44:32.280431 19047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:32 dist-test-slave-zfp8 krb5kdc[18807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:32.289424 19047 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:32.290990 19047 webserver.cc:460] Webserver started at http://127.15.102.194:42363/ using document root <none> and password file <none>
I20260620 18:44:32.291759 19047 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:32.291908 19047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:32.292219 19047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:32.295109 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "6a9a07eee7fc42fd909182380a266752"
format_stamp: "Formatted at 2026-06-20 18:44:32 on dist-test-slave-zfp8"
server_key: "faa00576187ca002a28232ffc5668e2e"
server_key_iv: "2a225f5005267194088844a710a53a18"
server_key_version: "encryptionkey@0"
I20260620 18:44:32.295948 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "6a9a07eee7fc42fd909182380a266752"
format_stamp: "Formatted at 2026-06-20 18:44:32 on dist-test-slave-zfp8"
server_key: "faa00576187ca002a28232ffc5668e2e"
server_key_iv: "2a225f5005267194088844a710a53a18"
server_key_version: "encryptionkey@0"
I20260620 18:44:32.301358 19047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:32.304898 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:32.306213 19047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:44:32.306392 19047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "6a9a07eee7fc42fd909182380a266752"
format_stamp: "Formatted at 2026-06-20 18:44:32 on dist-test-slave-zfp8"
server_key: "faa00576187ca002a28232ffc5668e2e"
server_key_iv: "2a225f5005267194088844a710a53a18"
server_key_version: "encryptionkey@0"
I20260620 18:44:32.306584 19047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:32.322391 19047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:32.326239 19047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:32.326553 19047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:32.327659 19047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:32.329027 19047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:32.329133 19047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:32.329294 19047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:32.329371 19047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:32.362620 19047 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:34107
I20260620 18:44:32.362669 19174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:34107 every 8 connection(s)
I20260620 18:44:32.363962 19047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:32.368860 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 19047
I20260620 18:44:32.369050 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:32.369483 15771 external_mini_cluster.cc:1428] Setting key d08a2f5c32568a2888a818d5ef4ca404
Jun 20 18:44:32 dist-test-slave-zfp8 krb5kdc[18807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:32.385478 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:32.366406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:52807 (local address 127.15.102.254:38453)
0620 18:44:32.366626 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:32.366632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:32.367872 (+ 1240us) server_negotiation.cc:407] Connection header received
0620 18:44:32.369718 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:32.369724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:32.369829 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:32.369948 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:32.371703 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.372430 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.373957 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.374372 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.374517 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:32.378534 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:32.378544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:32.378552 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:32.378608 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:32.382212 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.382924 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.382933 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.382939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.383096 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.383578 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.383586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.383593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.383970 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:32.384130 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:32.384789 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 18:44:32.385055 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:44:32.386418 19177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:32.366964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:38453 (local address 127.15.102.194:52807)
0620 18:44:32.367712 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:32.367751 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:32.369420 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:32.370211 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:32.370220 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:32.370865 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:32.371536 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:32.371554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.372589 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.372593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:32.373816 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:32.373826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.374634 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.374643 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:32.374987 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:32.375780 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:32.375824 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:32.378321 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:32.382355 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:32.382363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:32.382375 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:32.382780 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:32.383212 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:32.383215 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:32.383217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:32.383454 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:32.384097 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:32.384102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:32.384416 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 18:44:32.384630 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 18:44:32.385123 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":322,"threads_started":1}
I20260620 18:44:32.387956 19175 heartbeater.cc:347] Connected to a master server at 127.15.102.254:38453
I20260620 18:44:32.388296 19175 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
I20260620 18:44:32.389226 19175 heartbeater.cc:511] Master 127.15.102.254:38453 requested a full tablet report, sending...
I20260620 18:44:32.390722 18852 ts_manager.cc:194] Registered new tserver with Master: 6a9a07eee7fc42fd909182380a266752 (127.15.102.194:34107)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
I20260620 18:44:32.391567 18852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:52807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:32.413753 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38453
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:32.738850 19180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:32.739202 19180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:32.739388 19180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:32.746888 19180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:32.746996 19180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:32.747115 19180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:32.757191 19180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:32.758437 19180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:32.759796 19180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:32.772852 19188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:32.772855 19186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:32.772886 19185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:32.773662 19180 server_base.cc:1034] running on GCE node
I20260620 18:44:32.774220 19180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:32.775120 19180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:32.776353 19180 hybrid_clock.cc:648] HybridClock initialized: now 1781981072776330 us; error 72 us; skew 500 ppm
I20260620 18:44:32.776701 19180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:32 dist-test-slave-zfp8 krb5kdc[18807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:32.779888 19180 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:32.781359 19180 webserver.cc:460] Webserver started at http://127.15.102.195:34735/ using document root <none> and password file <none>
I20260620 18:44:32.782034 19180 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:32.782178 19180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:32.782487 19180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:32.785367 19180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "2ca6b21038f04ccd92c686307eb0a03f"
format_stamp: "Formatted at 2026-06-20 18:44:32 on dist-test-slave-zfp8"
server_key: "e35d7e22fc0c2ae614bd5f6c9c4c97b3"
server_key_iv: "6b6bba35148748e3830b8a0a531a1782"
server_key_version: "encryptionkey@0"
I20260620 18:44:32.786123 19180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "2ca6b21038f04ccd92c686307eb0a03f"
format_stamp: "Formatted at 2026-06-20 18:44:32 on dist-test-slave-zfp8"
server_key: "e35d7e22fc0c2ae614bd5f6c9c4c97b3"
server_key_iv: "6b6bba35148748e3830b8a0a531a1782"
server_key_version: "encryptionkey@0"
I20260620 18:44:32.791527 19180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:44:32.794839 19195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:32.796177 19180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:32.796340 19180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "2ca6b21038f04ccd92c686307eb0a03f"
format_stamp: "Formatted at 2026-06-20 18:44:32 on dist-test-slave-zfp8"
server_key: "e35d7e22fc0c2ae614bd5f6c9c4c97b3"
server_key_iv: "6b6bba35148748e3830b8a0a531a1782"
server_key_version: "encryptionkey@0"
I20260620 18:44:32.796538 19180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:32.813176 19180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:32.817327 19180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:32.817691 19180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:32.818675 19180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:32.820091 19180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:32.820202 19180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:32.820349 19180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:32.820418 19180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:32.853086 19180 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:46439
I20260620 18:44:32.853134 19307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:46439 every 8 connection(s)
I20260620 18:44:32.854373 19180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:32.861935 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 19180
I20260620 18:44:32.862217 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:32.862739 15771 external_mini_cluster.cc:1428] Setting key c9775408d62600cc3e977546b666bd99
Jun 20 18:44:32 dist-test-slave-zfp8 krb5kdc[18807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:32.875102 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:32.856943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:46077 (local address 127.15.102.254:38453)
0620 18:44:32.857113 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:32.857118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:32.858234 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 18:44:32.860068 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:32.860073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:32.860166 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:32.860278 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:32.861833 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.862812 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.864895 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.865303 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.865439 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:32.869231 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:32.869237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:32.869242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:32.869280 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:32.872496 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.873247 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.873252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.873254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.873366 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.873723 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.873726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.873728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.873979 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:32.874085 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:32.874713 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 18:44:32.874875 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:44:32.876084 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:32.857358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:38453 (local address 127.15.102.195:46077)
0620 18:44:32.858065 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:32.858102 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:32.859800 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:32.860508 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:32.860517 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:32.861021 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:32.861665 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:32.861684 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.862970 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.862982 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:32.864752 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:32.864764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.865530 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.865540 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:32.865797 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:32.866403 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:32.866431 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:32.869013 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:32.872645 (+ 3632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:32.872651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:32.872662 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:32.873073 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:32.873458 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:32.873461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:32.873464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:32.873624 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:32.874082 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:32.874088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:32.874398 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 18:44:32.874610 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 18:44:32.874964 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":259,"threads_started":1}
I20260620 18:44:32.877671 19308 heartbeater.cc:347] Connected to a master server at 127.15.102.254:38453
I20260620 18:44:32.877975 19308 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:32.878700 19308 heartbeater.cc:511] Master 127.15.102.254:38453 requested a full tablet report, sending...
I20260620 18:44:32.880158 18852 ts_manager.cc:194] Registered new tserver with Master: 2ca6b21038f04ccd92c686307eb0a03f (127.15.102.195:46439)
I20260620 18:44:32.880940 18852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:46077
I20260620 18:44:32.890172 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:44:32.898763 19041 heartbeater.cc:503] Master 127.15.102.254:38453 was elected leader, sending a full tablet report...
Jun 20 18:44:32 dist-test-slave-zfp8 krb5kdc[18807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 18:44:32 dist-test-slave-zfp8 krb5kdc[18807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:32.936691 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:32.923185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.15.102.254:38453)
0620 18:44:32.923449 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:32.923458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:32.923830 (+ 372us) server_negotiation.cc:407] Connection header received
0620 18:44:32.924054 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:32.924059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:32.924130 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:32.924245 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:32.925574 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.926325 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.927636 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.928028 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.928180 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:32.931859 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:32.931864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:32.931867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:32.931899 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:32.934138 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.934749 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.934758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.934760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.934886 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.935305 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.935309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.935311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.935607 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:32.935855 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:32.936165 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:44:32.936387 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260620 18:44:32.937243 18852 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39098
I20260620 18:44:32.939276 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18913
Jun 20 18:44:32 dist-test-slave-zfp8 krb5kdc[18807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:32.954576 19319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:32.940245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.15.102.193:45179)
0620 18:44:32.940907 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:32.940914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:32.940938 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:44:32.940991 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:32.940997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:32.941198 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:32.941321 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:32.942452 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.943451 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.944566 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.945098 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.945188 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:32.949247 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:32.949258 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:32.949271 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:32.949318 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:32.952169 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.952796 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.952802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.952804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.952913 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.953207 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.953210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.953212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.953539 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:32.953748 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:32.953980 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 18:44:32.954164 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":350,"threads_started":1}
W20260620 18:44:32.955857 19036 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38482
W20260620 18:44:32.956965 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18913: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:32.957218 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 18913
I20260620 18:44:32.973496 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19047
Jun 20 18:44:32 dist-test-slave-zfp8 krb5kdc[18807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:44:32.988628 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:32.974302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39608 (local address 127.15.102.194:34107)
0620 18:44:32.974848 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:32.974854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:32.974878 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:44:32.974934 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:32.974941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:32.975148 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:32.975351 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:32.976415 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.977407 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.978692 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:32.979226 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:32.979469 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:32.983409 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:32.983419 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:32.983434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:32.983465 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:32.986249 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.986861 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.986868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.986870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.986982 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:32.987376 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:32.987379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:32.987381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:32.987686 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:32.987896 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:32.988156 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 18:44:32.988352 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":295,"threads_started":1}
W20260620 18:44:32.989885 19170 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39608
W20260620 18:44:32.990944 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19047: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:32.991101 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 19047
I20260620 18:44:33.006673 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19180
Jun 20 18:44:33 dist-test-slave-zfp8 krb5kdc[18807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981072, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:44:33.022173 19321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:33.007603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.15.102.195:46439)
0620 18:44:33.008176 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:33.008182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:33.008212 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:44:33.008268 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:33.008274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:33.008479 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:33.008605 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:33.009832 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:33.010850 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:33.012124 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:33.012667 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:33.012766 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:33.016819 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:33.016830 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:33.016846 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:33.016879 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:33.019802 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:33.020426 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:33.020432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:33.020435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:33.020543 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:33.020823 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:33.020826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:33.020828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:33.021137 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:33.021368 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:33.021567 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 18:44:33.021747 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":257,"threads_started":1}
W20260620 18:44:33.023512 19303 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35870
W20260620 18:44:33.024650 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19180: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:44:33.024852 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 19180
I20260620 18:44:33.040376 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 18820
I20260620 18:44:33.041605 18882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:33.142138 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 18820
2026-06-20T18:44:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3525 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:33 dist-test-slave-zfp8 krb5kdc[19325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:33 dist-test-slave-zfp8 krb5kdc[19325](info): set up 2 sockets
Jun 20 18:44:33 dist-test-slave-zfp8 krb5kdc[19325](info): commencing operation
krb5kdc: starting...
W20260620 18:44:34.235901 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:34 dist-test-slave-zfp8 krb5kdc[19325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:34.427208 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:34105
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:34105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:34.756454 19338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:34.756744 19338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:34.756830 19338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:34.763978 19338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:34.764088 19338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:34.764146 19338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:34.764206 19338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:34.764249 19338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:34.773836 19338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:34105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:34.775179 19338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:34.776455 19338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:34.788586 19343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:34.789096 19344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:34.789110 19346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:34.790108 19338 server_base.cc:1034] running on GCE node
I20260620 18:44:34.790954 19338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:34.792507 19338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:34.793751 19338 hybrid_clock.cc:648] HybridClock initialized: now 1781981074793704 us; error 61 us; skew 500 ppm
I20260620 18:44:34.794106 19338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:34 dist-test-slave-zfp8 krb5kdc[19325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981074, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:34.797233 19338 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:34.798674 19338 webserver.cc:460] Webserver started at http://127.15.102.254:45345/ using document root <none> and password file <none>
I20260620 18:44:34.799453 19338 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:34.799592 19338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:34.799906 19338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:34.802832 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "5e09410770eb44bba37a595d95b05a57"
format_stamp: "Formatted at 2026-06-20 18:44:34 on dist-test-slave-zfp8"
server_key: "45f6e833a938751701529a469086d407"
server_key_iv: "d8d3c6ab80e00a1ef0ef9d47e7a92697"
server_key_version: "encryptionkey@0"
I20260620 18:44:34.803650 19338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "5e09410770eb44bba37a595d95b05a57"
format_stamp: "Formatted at 2026-06-20 18:44:34 on dist-test-slave-zfp8"
server_key: "45f6e833a938751701529a469086d407"
server_key_iv: "d8d3c6ab80e00a1ef0ef9d47e7a92697"
server_key_version: "encryptionkey@0"
I20260620 18:44:34.808660 19338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:44:34.812053 19353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:34.813390 19338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:34.813596 19338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "5e09410770eb44bba37a595d95b05a57"
format_stamp: "Formatted at 2026-06-20 18:44:34 on dist-test-slave-zfp8"
server_key: "45f6e833a938751701529a469086d407"
server_key_iv: "d8d3c6ab80e00a1ef0ef9d47e7a92697"
server_key_version: "encryptionkey@0"
I20260620 18:44:34.813812 19338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:34.835668 19338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:34.839674 19338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:34.839973 19338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:34.858315 19338 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:34105
I20260620 18:44:34.858341 19404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:34105 every 8 connection(s)
I20260620 18:44:34.859688 19338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:34.863336 19405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:34.866539 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 19338
I20260620 18:44:34.866787 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:34.867411 15771 external_mini_cluster.cc:1428] Setting key 6fdcc21983125f3d2b78b06cbaacfe2d
I20260620 18:44:34.876056 19405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57: Bootstrap starting.
I20260620 18:44:34.879839 19405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:34.881158 19405 log.cc:824] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:44:34 dist-test-slave-zfp8 krb5kdc[19325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:34.884292 19405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57: No bootstrap required, opened a new log
I20260620 18:44:34.889941 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:34.869387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.15.102.254:34105)
0620 18:44:34.870176 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:34.870188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:34.870232 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:44:34.871885 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:34.871908 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:34.872341 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:34.872680 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:34.874464 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:34.875818 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:34.877534 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:34.878211 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:34.878334 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:34.882833 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:34.882840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:34.882855 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:34.882892 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:34.885898 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:34.886581 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:34.886598 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:34.886604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:34.886764 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:34.887141 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:34.887146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:34.887150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:34.887787 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:34.888072 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:34.888547 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 18:44:34.888950 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":377,"threads_started":1}
I20260620 18:44:34.892167 19405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09410770eb44bba37a595d95b05a57" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34105 } }
I20260620 18:44:34.892544 19405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:34.892673 19405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e09410770eb44bba37a595d95b05a57, State: Initialized, Role: FOLLOWER
I20260620 18:44:34.893188 19405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09410770eb44bba37a595d95b05a57" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34105 } }
I20260620 18:44:34.893443 19405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:34.893581 19405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:34.893757 19405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:34.895921 19405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09410770eb44bba37a595d95b05a57" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34105 } }
I20260620 18:44:34.896399 19405 leader_election.cc:302] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e09410770eb44bba37a595d95b05a57; no voters:
I20260620 18:44:34.896929 19405 leader_election.cc:288] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:34.897068 19410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:34.897759 19410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [term 1 LEADER]: Becoming Leader. State: Replica: 5e09410770eb44bba37a595d95b05a57, State: Running, Role: LEADER
I20260620 18:44:34.898315 19410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09410770eb44bba37a595d95b05a57" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34105 } }
I20260620 18:44:34.898964 19405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:34.901309 19412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e09410770eb44bba37a595d95b05a57. Latest consensus state: current_term: 1 leader_uuid: "5e09410770eb44bba37a595d95b05a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09410770eb44bba37a595d95b05a57" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34105 } } }
I20260620 18:44:34.901284 19411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e09410770eb44bba37a595d95b05a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e09410770eb44bba37a595d95b05a57" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34105 } } }
I20260620 18:44:34.901722 19412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:34.901760 19411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:34.903213 19416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:34.908257 19416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:34.917626 19416 catalog_manager.cc:1347] Generated new cluster ID: 2a1230c3c9db48469c7979ef9e625bcf
I20260620 18:44:34.917764 19416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:34.931310 19416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:34.933094 19416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:34.940850 19416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e09410770eb44bba37a595d95b05a57: Generated new TSK 0
I20260620 18:44:34.941834 19416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:34.992687 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34105
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:35.322139 19431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:35.322438 19431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:35.322589 19431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:35.329873 19431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:35.329985 19431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:35.330188 19431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:35.339890 19431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:35.341195 19431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:35.342525 19431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:35.355257 19436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:35.355299 19437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:35.355417 19439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:35.356626 19431 server_base.cc:1034] running on GCE node
I20260620 18:44:35.357300 19431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:35.358264 19431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:35.359485 19431 hybrid_clock.cc:648] HybridClock initialized: now 1781981075359443 us; error 58 us; skew 500 ppm
I20260620 18:44:35.359802 19431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:35 dist-test-slave-zfp8 krb5kdc[19325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981075, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:35.362946 19431 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:35.364455 19431 webserver.cc:460] Webserver started at http://127.15.102.193:42233/ using document root <none> and password file <none>
I20260620 18:44:35.365178 19431 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:35.365331 19431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:35.365638 19431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:35.368531 19431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "25d30017ca6f4f028026bcf3ae6edc40"
format_stamp: "Formatted at 2026-06-20 18:44:35 on dist-test-slave-zfp8"
server_key: "baf76a5709f0e7fea23363cb03fd0de6"
server_key_iv: "8a9097974b7f6095f2e269cd376b5de4"
server_key_version: "encryptionkey@0"
I20260620 18:44:35.369305 19431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "25d30017ca6f4f028026bcf3ae6edc40"
format_stamp: "Formatted at 2026-06-20 18:44:35 on dist-test-slave-zfp8"
server_key: "baf76a5709f0e7fea23363cb03fd0de6"
server_key_iv: "8a9097974b7f6095f2e269cd376b5de4"
server_key_version: "encryptionkey@0"
I20260620 18:44:35.374276 19431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:44:35.377624 19446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:35.378983 19431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:35.379398 19431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "25d30017ca6f4f028026bcf3ae6edc40"
format_stamp: "Formatted at 2026-06-20 18:44:35 on dist-test-slave-zfp8"
server_key: "baf76a5709f0e7fea23363cb03fd0de6"
server_key_iv: "8a9097974b7f6095f2e269cd376b5de4"
server_key_version: "encryptionkey@0"
I20260620 18:44:35.379639 19431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:35.400367 19431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:35.404558 19431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:35.404865 19431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:35.405815 19431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:35.407146 19431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:35.407274 19431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:35.407431 19431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:35.407506 19431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:35.440964 19431 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:45207
I20260620 18:44:35.440984 19558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:45207 every 8 connection(s)
I20260620 18:44:35.442226 19431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:35.444015 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 19431
I20260620 18:44:35.444227 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:35.444967 15771 external_mini_cluster.cc:1428] Setting key 90dd407d23dacdd4881949e129d727cc
Jun 20 18:44:35 dist-test-slave-zfp8 krb5kdc[19325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981075, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:44:35.466929 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:35.444865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:41155 (local address 127.15.102.254:34105)
0620 18:44:35.450868 (+ 6003us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:35.450873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:35.450890 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:44:35.450938 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:35.450942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:35.451006 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:35.451100 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:35.453210 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:35.454219 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:35.455881 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:35.456600 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:35.456748 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:35.460338 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:35.460347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:35.460354 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:35.460400 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:35.464021 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:35.464718 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:35.464727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:35.464732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:35.464894 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:35.465300 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:35.465305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:35.465308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:35.465663 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:35.465833 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:35.466473 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 18:44:35.466664 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5889,"thread_start_us":5762,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
I20260620 18:44:35.468109 19562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:35.445616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34105 (local address 127.15.102.193:41155)
0620 18:44:35.446523 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:35.446579 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:35.449205 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:35.451442 (+ 2237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:35.451457 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:35.452275 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:35.453026 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:35.453046 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:35.454362 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:35.454366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:35.455730 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:35.455742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:35.456817 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:35.456822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:35.457032 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:35.457720 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:35.457746 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:35.460088 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:35.464156 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:35.464162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:35.464175 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:35.464577 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:35.465003 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:35.465007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:35.465009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:35.465177 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:35.465783 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:35.465789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:35.466115 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 18:44:35.466342 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 18:44:35.466829 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":355,"threads_started":1}
I20260620 18:44:35.469715 19559 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34105
I20260620 18:44:35.470054 19559 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:35.470938 19559 heartbeater.cc:511] Master 127.15.102.254:34105 requested a full tablet report, sending...
I20260620 18:44:35.473232 19370 ts_manager.cc:194] Registered new tserver with Master: 25d30017ca6f4f028026bcf3ae6edc40 (127.15.102.193:45207)
I20260620 18:44:35.474844 19370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:41155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:35.492501 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34105
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:35.809021 19565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:35.809337 19565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:35.809505 19565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:35.816840 19565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:35.817000 19565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:35.817168 19565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:35.827014 19565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:35.828338 19565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:35.829672 19565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:35.842790 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:35.842744 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:35.842836 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:35.843663 19565 server_base.cc:1034] running on GCE node
I20260620 18:44:35.844321 19565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:35.845156 19565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:35.846379 19565 hybrid_clock.cc:648] HybridClock initialized: now 1781981075846347 us; error 46 us; skew 500 ppm
I20260620 18:44:35.846681 19565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:35 dist-test-slave-zfp8 krb5kdc[19325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981075, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:35.849807 19565 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:35.851281 19565 webserver.cc:460] Webserver started at http://127.15.102.194:38111/ using document root <none> and password file <none>
I20260620 18:44:35.851933 19565 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:35.852028 19565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:35.852272 19565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:35.855043 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "f0125c1cca3541f9b2c0e0e144dc3492"
format_stamp: "Formatted at 2026-06-20 18:44:35 on dist-test-slave-zfp8"
server_key: "1ae1bc1c6d3246489cdbe64e347ad6e4"
server_key_iv: "9105162467c574fac30e84dba5b96487"
server_key_version: "encryptionkey@0"
I20260620 18:44:35.855847 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "f0125c1cca3541f9b2c0e0e144dc3492"
format_stamp: "Formatted at 2026-06-20 18:44:35 on dist-test-slave-zfp8"
server_key: "1ae1bc1c6d3246489cdbe64e347ad6e4"
server_key_iv: "9105162467c574fac30e84dba5b96487"
server_key_version: "encryptionkey@0"
I20260620 18:44:35.861090 19565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:35.864538 19580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:35.865929 19565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:44:35.866101 19565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "f0125c1cca3541f9b2c0e0e144dc3492"
format_stamp: "Formatted at 2026-06-20 18:44:35 on dist-test-slave-zfp8"
server_key: "1ae1bc1c6d3246489cdbe64e347ad6e4"
server_key_iv: "9105162467c574fac30e84dba5b96487"
server_key_version: "encryptionkey@0"
I20260620 18:44:35.866307 19565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:35.877846 19565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:35.881883 19565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:35.882109 19565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:35.883070 19565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:35.884426 19565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:35.884496 19565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:35.884608 19565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:35.884666 19565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:35.915757 19565 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:40435
I20260620 18:44:35.915828 19692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:40435 every 8 connection(s)
I20260620 18:44:35.917127 19565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:35.920271 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 19565
I20260620 18:44:35.920468 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:35.920868 15771 external_mini_cluster.cc:1428] Setting key 30cb963647186c62b6f1cc641e50fcce
Jun 20 18:44:35 dist-test-slave-zfp8 krb5kdc[19325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981075, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:35.938078 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:35.919811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:58835 (local address 127.15.102.254:34105)
0620 18:44:35.920043 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:35.920051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:35.921594 (+ 1543us) server_negotiation.cc:407] Connection header received
0620 18:44:35.923113 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:35.923119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:35.923205 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:35.923338 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:35.924717 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:35.925847 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:35.927373 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:35.927926 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:35.928022 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:35.932360 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:35.932365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:35.932368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:35.932397 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:35.934857 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:35.935828 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:35.935832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:35.935834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:35.935952 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:35.936436 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:35.936438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:35.936440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:35.936668 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:35.936769 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:35.937688 (+ 919us) server_negotiation.cc:299] Negotiation successful
0620 18:44:35.937845 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:44:35.939744 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:35.920325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34105 (local address 127.15.102.194:58835)
0620 18:44:35.921180 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:35.921232 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:35.922861 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:35.923512 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:35.923520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:35.924010 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:35.924555 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:35.924571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:35.925986 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:35.925989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:35.927189 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:35.927200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:35.928152 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:35.928161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:35.928468 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:35.929204 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:35.929238 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:35.932139 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:35.934994 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:35.935005 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:35.935026 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:35.935710 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:35.936061 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:35.936069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:35.936076 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:35.936321 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:35.936800 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:35.936812 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:35.937294 (+ 482us) client_negotiation.cc:769] Sending connection context
0620 18:44:35.937608 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 18:44:35.938180 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
I20260620 18:44:35.941215 19693 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34105
I20260620 18:44:35.941488 19693 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:35.942234 19693 heartbeater.cc:511] Master 127.15.102.254:34105 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
I20260620 18:44:35.943805 19370 ts_manager.cc:194] Registered new tserver with Master: f0125c1cca3541f9b2c0e0e144dc3492 (127.15.102.194:40435)
I20260620 18:44:35.944571 19370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:58835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:35.964509 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34105
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:36.281592 19698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:36.281862 19698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:36.282037 19698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:36.289469 19698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:36.289574 19698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:36.289744 19698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:36.299326 19698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:36.300614 19698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:36.301918 19698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:36.315099 19706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:36.315066 19703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:36.315152 19704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:36.315809 19698 server_base.cc:1034] running on GCE node
I20260620 18:44:36.316457 19698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:36.317368 19698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:36.318591 19698 hybrid_clock.cc:648] HybridClock initialized: now 1781981076318541 us; error 33 us; skew 500 ppm
I20260620 18:44:36.318933 19698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:36 dist-test-slave-zfp8 krb5kdc[19325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:36.322136 19698 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:36.323683 19698 webserver.cc:460] Webserver started at http://127.15.102.195:46037/ using document root <none> and password file <none>
I20260620 18:44:36.324357 19698 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:36.324499 19698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:36.324810 19698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:36.327731 19698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "0ba3cd59b85b4aa7a41224005948bfd9"
format_stamp: "Formatted at 2026-06-20 18:44:36 on dist-test-slave-zfp8"
server_key: "487f48de4f24a651986307bff5586439"
server_key_iv: "fb88c7117a5c989206f70dff7ce88b90"
server_key_version: "encryptionkey@0"
I20260620 18:44:36.328500 19698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "0ba3cd59b85b4aa7a41224005948bfd9"
format_stamp: "Formatted at 2026-06-20 18:44:36 on dist-test-slave-zfp8"
server_key: "487f48de4f24a651986307bff5586439"
server_key_iv: "fb88c7117a5c989206f70dff7ce88b90"
server_key_version: "encryptionkey@0"
I20260620 18:44:36.333514 19698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:36.336894 19713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:36.338260 19698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:44:36.338459 19698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "0ba3cd59b85b4aa7a41224005948bfd9"
format_stamp: "Formatted at 2026-06-20 18:44:36 on dist-test-slave-zfp8"
server_key: "487f48de4f24a651986307bff5586439"
server_key_iv: "fb88c7117a5c989206f70dff7ce88b90"
server_key_version: "encryptionkey@0"
I20260620 18:44:36.338688 19698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:36.357702 19698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:36.362411 19698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:36.362735 19698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:36.363898 19698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:36.365422 19698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:36.365533 19698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:36.365703 19698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:36.365787 19698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:36.400538 19698 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:42247
I20260620 18:44:36.400612 19825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:42247 every 8 connection(s)
I20260620 18:44:36.402040 19698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:36.412400 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 19698
I20260620 18:44:36.412613 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:36.413268 15771 external_mini_cluster.cc:1428] Setting key 625562f4650e8c7bb2492d95df724e13
Jun 20 18:44:36 dist-test-slave-zfp8 krb5kdc[19325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:36.422336 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:36.404538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:37593 (local address 127.15.102.254:34105)
0620 18:44:36.404727 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:36.404732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:36.405942 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 18:44:36.407844 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:36.407849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:36.407938 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:36.408030 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:36.409552 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.410284 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.412076 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.412616 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.412775 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:36.416918 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:36.416922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:36.416925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:36.416956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:36.419775 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.420448 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.420452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.420454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.420562 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.420931 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.420934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.420935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.421169 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:36.421275 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:36.421940 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 18:44:36.422085 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:44:36.423411 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:36.405055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34105 (local address 127.15.102.195:37593)
0620 18:44:36.405778 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:36.405817 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:36.407577 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:36.408253 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:36.408261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:36.408797 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:36.409373 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:36.409391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.410424 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.410427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:36.411889 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:36.411926 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.412902 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.412911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:36.413232 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:36.413964 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:36.413996 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:36.416746 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:36.419893 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:36.419900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:36.419912 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:36.420339 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:36.420653 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:36.420656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:36.420658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:36.420832 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:36.421271 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:36.421277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:36.421592 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 18:44:36.421837 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 18:44:36.422209 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":279,"threads_started":1}
I20260620 18:44:36.424839 19826 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34105
I20260620 18:44:36.425154 19826 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:36.425899 19826 heartbeater.cc:511] Master 127.15.102.254:34105 requested a full tablet report, sending...
I20260620 18:44:36.427428 19370 ts_manager.cc:194] Registered new tserver with Master: 0ba3cd59b85b4aa7a41224005948bfd9 (127.15.102.195:42247)
I20260620 18:44:36.428226 19370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:37593
I20260620 18:44:36.428979 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:44:36 dist-test-slave-zfp8 krb5kdc[19325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:44:36 dist-test-slave-zfp8 krb5kdc[19325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:36.478438 19559 heartbeater.cc:503] Master 127.15.102.254:34105 was elected leader, sending a full tablet report...
I20260620 18:44:36.479509 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:36.464073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.15.102.193:45207)
0620 18:44:36.464718 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:36.464728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:36.464774 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:44:36.464850 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:36.464863 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:36.465078 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:36.465245 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:36.466687 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.467747 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.469024 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.469591 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.469729 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:36.473748 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:36.473760 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:36.473781 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:36.473825 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:36.476573 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.477213 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.477220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.477222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.477335 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.477654 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.477657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.477659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.477976 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:36.478201 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:36.478635 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 18:44:36.478956 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":311,"threads_started":1}
Jun 20 18:44:36 dist-test-slave-zfp8 krb5kdc[19325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:36.497939 19561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:36.484175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.15.102.254:34105)
0620 18:44:36.484391 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:36.484398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:36.484576 (+ 178us) server_negotiation.cc:407] Connection header received
0620 18:44:36.484720 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:36.484723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:36.484782 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:36.484865 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:36.486057 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.486745 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.488243 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.488657 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.488756 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:36.492768 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:36.492773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:36.492776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:36.492818 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:36.495423 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.496047 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.496059 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.496061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.496178 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.496534 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.496540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.496543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.496866 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:36.497106 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:36.497410 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:44:36.497614 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 18:44:36.498593 19369 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54636
I20260620 18:44:36.499782 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19431
I20260620 18:44:36.512225 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:36.500679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.15.102.193:45207)
0620 18:44:36.500896 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:36.500906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:36.501167 (+ 261us) server_negotiation.cc:407] Connection header received
0620 18:44:36.501305 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:36.501310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:36.501402 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:36.501480 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:36.502663 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.503463 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.504615 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.504998 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.505085 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:36.507190 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:36.507194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:36.507198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:36.507226 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:36.509862 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.510435 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.510439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.510441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.510568 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.510873 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.510876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.510878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.511103 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:36.511355 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:36.511684 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 18:44:36.511888 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:44:36.512770 19554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:36.594560 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 19431
I20260620 18:44:36.610643 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19565
Jun 20 18:44:36 dist-test-slave-zfp8 krb5kdc[19325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:44:36.626323 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:36.611535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.15.102.194:40435)
0620 18:44:36.612102 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:36.612109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:36.612133 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:44:36.612190 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:36.612196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:36.612436 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:36.612600 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:36.613840 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.614849 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.616176 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.616779 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.616911 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:36.620888 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:36.620898 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:36.620913 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:36.620944 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:36.623884 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.624519 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.624527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.624529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.624650 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.624964 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.624967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.624969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.625286 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:36.625501 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:36.625847 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:44:36.626029 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":272,"threads_started":1}
I20260620 18:44:36.627725 19688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:36.714845 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 19565
I20260620 18:44:36.731164 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19698
Jun 20 18:44:36 dist-test-slave-zfp8 krb5kdc[19325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:44:36.747157 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:36.732247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.15.102.195:42247)
0620 18:44:36.732901 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:36.732907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:36.732937 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:44:36.732998 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:36.733004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:36.733208 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:36.733341 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:36.734560 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.735678 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.736960 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:36.737502 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:36.737599 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:36.741934 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:36.741946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:36.741960 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:36.741997 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:36.744807 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.745379 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.745386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.745388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.745500 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:36.745815 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:36.745817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:36.745819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:36.746129 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:36.746347 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:36.746574 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:44:36.746770 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":347,"threads_started":1}
I20260620 18:44:36.748536 19821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:36.830921 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 19698
I20260620 18:44:36.846763 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19338
I20260620 18:44:36.847952 19400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:36.949586 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 19338
2026-06-20T18:44:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3808 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:37 dist-test-slave-zfp8 krb5kdc[19845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:37 dist-test-slave-zfp8 krb5kdc[19845](info): set up 2 sockets
Jun 20 18:44:37 dist-test-slave-zfp8 krb5kdc[19845](info): commencing operation
krb5kdc: starting...
W20260620 18:44:38.035804 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:38 dist-test-slave-zfp8 krb5kdc[19845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:38.228993 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46863
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:38.561087 19858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:38.561471 19858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:38.561655 19858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:38.568892 19858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:38.569065 19858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:38.569178 19858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:38.569294 19858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:38.569380 19858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:38.579300 19858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:38.580709 19858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:38.582048 19858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:38.594074 19866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:38.594111 19864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:38.594156 19863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:38.596684 19858 server_base.cc:1034] running on GCE node
I20260620 18:44:38.597571 19858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:38.598996 19858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:38.600232 19858 hybrid_clock.cc:648] HybridClock initialized: now 1781981078600189 us; error 41 us; skew 500 ppm
I20260620 18:44:38.600595 19858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:38 dist-test-slave-zfp8 krb5kdc[19845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981078, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:38.608058 19858 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:38.609395 19858 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:38.609541 19858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:38.609875 19858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:38.612995 19858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "f2acaedc505a4168b86181b60606d547"
format_stamp: "Formatted at 2026-06-20 18:44:38 on dist-test-slave-zfp8"
server_key: "8a5406aac0b71ec79cb0fdbe7712c9d7"
server_key_iv: "2a5e17e382be206bf353e57aff048663"
server_key_version: "encryptionkey@0"
I20260620 18:44:38.613817 19858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "f2acaedc505a4168b86181b60606d547"
format_stamp: "Formatted at 2026-06-20 18:44:38 on dist-test-slave-zfp8"
server_key: "8a5406aac0b71ec79cb0fdbe7712c9d7"
server_key_iv: "2a5e17e382be206bf353e57aff048663"
server_key_version: "encryptionkey@0"
I20260620 18:44:38.618852 19858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:44:38.622267 19872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:38.623757 19858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:38.623946 19858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "f2acaedc505a4168b86181b60606d547"
format_stamp: "Formatted at 2026-06-20 18:44:38 on dist-test-slave-zfp8"
server_key: "8a5406aac0b71ec79cb0fdbe7712c9d7"
server_key_iv: "2a5e17e382be206bf353e57aff048663"
server_key_version: "encryptionkey@0"
I20260620 18:44:38.624166 19858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:38.644410 19858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:38.648422 19858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:38.648692 19858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:38.667114 19858 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:46863
I20260620 18:44:38.667106 19923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:46863 every 8 connection(s)
I20260620 18:44:38.668067 19858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:38.668637 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 19858
I20260620 18:44:38.669049 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:38.669806 15771 external_mini_cluster.cc:1428] Setting key a07e2c80ea9d34edb69ad7945d38e3fd
I20260620 18:44:38.672492 19924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:38.682679 19924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547: Bootstrap starting.
Jun 20 18:44:38 dist-test-slave-zfp8 krb5kdc[19845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:38.686091 19924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:38.687157 19924 log.cc:824] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:38.690361 19924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547: No bootstrap required, opened a new log
I20260620 18:44:38.691567 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:38.671867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.15.102.254:46863)
0620 18:44:38.672831 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:38.672843 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:38.672883 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:44:38.674484 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:38.674505 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:38.674909 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:38.675344 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:38.676848 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:38.678158 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:38.679506 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:38.680204 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:38.680410 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:38.684316 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:38.684324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:38.684339 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:38.684377 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:38.687570 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:38.688354 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:38.688363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:38.688372 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:38.688539 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:38.688917 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:38.688920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:38.688922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:38.689441 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:38.689645 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:38.690088 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 18:44:38.690449 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":506,"threads_started":1}
I20260620 18:44:38.698535 19924 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2acaedc505a4168b86181b60606d547" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46863 } }
I20260620 18:44:38.698909 19924 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:38.699039 19924 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2acaedc505a4168b86181b60606d547, State: Initialized, Role: FOLLOWER
I20260620 18:44:38.699654 19924 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2acaedc505a4168b86181b60606d547" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46863 } }
I20260620 18:44:38.699891 19924 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:38.700028 19924 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:38.700205 19924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:38.702315 19924 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2acaedc505a4168b86181b60606d547" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46863 } }
I20260620 18:44:38.702804 19924 leader_election.cc:302] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2acaedc505a4168b86181b60606d547; no voters:
I20260620 18:44:38.703459 19924 leader_election.cc:288] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:38.703593 19929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:38.704293 19929 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [term 1 LEADER]: Becoming Leader. State: Replica: f2acaedc505a4168b86181b60606d547, State: Running, Role: LEADER
I20260620 18:44:38.704828 19929 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2acaedc505a4168b86181b60606d547" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46863 } }
I20260620 18:44:38.705736 19924 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:38.707926 19931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2acaedc505a4168b86181b60606d547. Latest consensus state: current_term: 1 leader_uuid: "f2acaedc505a4168b86181b60606d547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2acaedc505a4168b86181b60606d547" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46863 } } }
I20260620 18:44:38.708345 19931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:38.709296 19930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2acaedc505a4168b86181b60606d547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2acaedc505a4168b86181b60606d547" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46863 } } }
I20260620 18:44:38.709673 19930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:38.712137 19937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:38.717110 19937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:38.728341 19937 catalog_manager.cc:1347] Generated new cluster ID: 1e0330f679644de2a1b26577c2da001a
I20260620 18:44:38.728530 19937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:38.760799 19937 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:38.762308 19937 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:38.768831 19937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547: Generated new TSK 0
I20260620 18:44:38.769737 19937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:38.818686 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46863
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:39.142177 19950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:39.142510 19950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:39.142689 19950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:39.149923 19950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:39.150084 19950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:39.150267 19950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:39.160038 19950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:39.161371 19950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:39.162700 19950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:39.175709 19956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:39.175782 19958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:39.175747 19955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:39.176513 19950 server_base.cc:1034] running on GCE node
I20260620 18:44:39.177150 19950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:39.178205 19950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:39.179433 19950 hybrid_clock.cc:648] HybridClock initialized: now 1781981079179379 us; error 51 us; skew 500 ppm
I20260620 18:44:39.179761 19950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:39 dist-test-slave-zfp8 krb5kdc[19845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981079, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:39.182855 19950 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:39.184015 19950 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:39.184108 19950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:39.184365 19950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:39.187160 19950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "4d18460526c641ae9054efae46b3bae6"
format_stamp: "Formatted at 2026-06-20 18:44:39 on dist-test-slave-zfp8"
server_key: "ab8049511654e70fc5e9b28310d1ae86"
server_key_iv: "bde48b2c964f6705c555365a025ec6e5"
server_key_version: "encryptionkey@0"
I20260620 18:44:39.188040 19950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "4d18460526c641ae9054efae46b3bae6"
format_stamp: "Formatted at 2026-06-20 18:44:39 on dist-test-slave-zfp8"
server_key: "ab8049511654e70fc5e9b28310d1ae86"
server_key_iv: "bde48b2c964f6705c555365a025ec6e5"
server_key_version: "encryptionkey@0"
I20260620 18:44:39.193106 19950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:39.196751 19964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:39.198210 19950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:44:39.198383 19950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "4d18460526c641ae9054efae46b3bae6"
format_stamp: "Formatted at 2026-06-20 18:44:39 on dist-test-slave-zfp8"
server_key: "ab8049511654e70fc5e9b28310d1ae86"
server_key_iv: "bde48b2c964f6705c555365a025ec6e5"
server_key_version: "encryptionkey@0"
I20260620 18:44:39.198592 19950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:39.237844 19950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:39.242090 19950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:39.242374 19950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:39.243414 19950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:39.244722 19950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:39.244836 19950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:39.244962 19950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:39.245064 19950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:39.278070 19950 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:34897
I20260620 18:44:39.278162 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:34897 every 8 connection(s)
I20260620 18:44:39.278980 19950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:39.279536 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 19950
I20260620 18:44:39.279760 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:39.280534 15771 external_mini_cluster.cc:1428] Setting key 81aa637b3c7ecd25efc398a93afb84ac
Jun 20 18:44:39 dist-test-slave-zfp8 krb5kdc[19845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981079, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:44:39.300081 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:39.281580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:56571 (local address 127.15.102.254:46863)
0620 18:44:39.282016 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:39.282023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:39.283226 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 18:44:39.285017 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:39.285025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:39.285117 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:39.285236 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:39.287317 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:39.288273 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:39.290130 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:39.290583 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:39.290719 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:39.294353 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:39.294358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:39.294361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:39.294394 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:39.297012 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:39.297746 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:39.297752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:39.297756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:39.297875 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:39.298309 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:39.298315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:39.298318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:39.298584 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:39.298755 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:39.299500 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 18:44:39.299718 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
I20260620 18:44:39.301165 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:39.282232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46863 (local address 127.15.102.193:56571)
0620 18:44:39.283054 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:39.283106 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:39.284757 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:39.285492 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:39.285504 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:39.286281 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:39.287108 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:39.287135 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:39.288431 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:39.288437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:39.289954 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:39.289966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:39.290830 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:39.290837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:39.291068 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:39.291880 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:39.291904 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:39.294114 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:39.297163 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:39.297172 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:39.297195 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:39.297602 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:39.297986 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:39.297991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:39.297995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:39.298164 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:39.298713 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:39.298720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:39.299069 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 18:44:39.299390 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 18:44:39.299901 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":330,"threads_started":1}
I20260620 18:44:39.302903 20077 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46863
I20260620 18:44:39.303220 20077 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:39.303936 20077 heartbeater.cc:511] Master 127.15.102.254:46863 requested a full tablet report, sending...
I20260620 18:44:39.306128 19889 ts_manager.cc:194] Registered new tserver with Master: 4d18460526c641ae9054efae46b3bae6 (127.15.102.193:34897)
I20260620 18:44:39.307667 19889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:56571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:39.324165 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46863
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:39.642446 20083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:39.642717 20083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:39.642881 20083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:39.649998 20083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:39.650105 20083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:39.650277 20083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:39.659969 20083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:39.661409 20083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:39.662719 20083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:39.676419 20088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:39.676401 20089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:39.677202 20083 server_base.cc:1034] running on GCE node
W20260620 18:44:39.677466 20091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:39.678177 20083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:39.679155 20083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:39.680588 20083 hybrid_clock.cc:648] HybridClock initialized: now 1781981079680539 us; error 41 us; skew 500 ppm
I20260620 18:44:39.680913 20083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:39 dist-test-slave-zfp8 krb5kdc[19845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981079, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:39.684273 20083 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:39.685485 20083 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:39.685621 20083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:39.685954 20083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:39.688886 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "2be73fd67e364d7ba9cbca065fb3a581"
format_stamp: "Formatted at 2026-06-20 18:44:39 on dist-test-slave-zfp8"
server_key: "1bc9edd009ea58db33714e9f63aaf4eb"
server_key_iv: "cd3275914c85c1c3bbcfd1c6ac85f93f"
server_key_version: "encryptionkey@0"
I20260620 18:44:39.689687 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "2be73fd67e364d7ba9cbca065fb3a581"
format_stamp: "Formatted at 2026-06-20 18:44:39 on dist-test-slave-zfp8"
server_key: "1bc9edd009ea58db33714e9f63aaf4eb"
server_key_iv: "cd3275914c85c1c3bbcfd1c6ac85f93f"
server_key_version: "encryptionkey@0"
I20260620 18:44:39.694933 20083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:44:39.698632 20097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:39.700013 20083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:44:39.700191 20083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "2be73fd67e364d7ba9cbca065fb3a581"
format_stamp: "Formatted at 2026-06-20 18:44:39 on dist-test-slave-zfp8"
server_key: "1bc9edd009ea58db33714e9f63aaf4eb"
server_key_iv: "cd3275914c85c1c3bbcfd1c6ac85f93f"
server_key_version: "encryptionkey@0"
I20260620 18:44:39.700409 20083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:39.721989 20083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:39.726506 20083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:39.726814 20083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:39.727938 20083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:39.729424 20083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:39.729543 20083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:39.729683 20083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:39.729772 20083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:39.762771 20083 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:40485
I20260620 18:44:39.762832 20209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:40485 every 8 connection(s)
I20260620 18:44:39.763695 20083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:39.773440 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 20083
I20260620 18:44:39.773667 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:39.774088 15771 external_mini_cluster.cc:1428] Setting key 31e3c7fa23c072f1195b64b54980dec1
Jun 20 18:44:39 dist-test-slave-zfp8 krb5kdc[19845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981079, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:39.783205 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:39.766197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:48099 (local address 127.15.102.254:46863)
0620 18:44:39.766373 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:39.766378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:39.767530 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 18:44:39.769312 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:39.769319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:39.769413 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:39.769526 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:39.771025 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:39.771763 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:39.773316 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:39.773844 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:39.773967 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:39.777617 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:39.777623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:39.777626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:39.777657 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:39.780188 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:39.780940 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:39.780947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:39.780951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:39.781081 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:39.781506 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:39.781511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:39.781514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:39.781761 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:39.781932 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:39.782621 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 18:44:39.782848 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:44:39.784267 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:39.766601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46863 (local address 127.15.102.194:48099)
0620 18:44:39.767364 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:39.767404 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:39.769034 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:39.769748 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:39.769756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:39.770257 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:39.770835 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:39.770851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:39.771925 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:39.771930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:39.773163 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:39.773175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:39.774071 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:39.774078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:39.774355 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:39.775040 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:39.775063 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:39.777378 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:39.780343 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:39.780351 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:39.780370 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:39.780814 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:39.781191 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:39.781197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:39.781201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:39.781385 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:39.781895 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:39.781906 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:39.782237 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 18:44:39.782520 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 18:44:39.783058 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":278,"threads_started":1}
I20260620 18:44:39.785835 20210 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46863
I20260620 18:44:39.786161 20210 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:39.786811 20210 heartbeater.cc:511] Master 127.15.102.254:46863 requested a full tablet report, sending...
I20260620 18:44:39.788322 19889 ts_manager.cc:194] Registered new tserver with Master: 2be73fd67e364d7ba9cbca065fb3a581 (127.15.102.194:40485)
I20260620 18:44:39.789552 19889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:48099
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:39.816385 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46863
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:40.135731 20215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:40.136018 20215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:40.136189 20215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:40.143332 20215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:40.143438 20215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:40.143658 20215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:40.153282 20215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:40.154686 20215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:40.156106 20215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:40.168865 20223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:40.168805 20221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:40.168896 20220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:40.169437 20215 server_base.cc:1034] running on GCE node
I20260620 18:44:40.170125 20215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:40.171036 20215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:40.172271 20215 hybrid_clock.cc:648] HybridClock initialized: now 1781981080172197 us; error 66 us; skew 500 ppm
I20260620 18:44:40.172619 20215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:40 dist-test-slave-zfp8 krb5kdc[19845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981080, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:40.175846 20215 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:40.177000 20215 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:40.177132 20215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:40.177466 20215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:40.180419 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "6903cbdd40d548e99ac2d22fba404cb8"
format_stamp: "Formatted at 2026-06-20 18:44:40 on dist-test-slave-zfp8"
server_key: "871fc65df925f7cb678894e845e9179c"
server_key_iv: "9d3ab1dced12f9fddfd90171b3708761"
server_key_version: "encryptionkey@0"
I20260620 18:44:40.181219 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "6903cbdd40d548e99ac2d22fba404cb8"
format_stamp: "Formatted at 2026-06-20 18:44:40 on dist-test-slave-zfp8"
server_key: "871fc65df925f7cb678894e845e9179c"
server_key_iv: "9d3ab1dced12f9fddfd90171b3708761"
server_key_version: "encryptionkey@0"
I20260620 18:44:40.186237 20215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:44:40.189675 20229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:40.191068 20215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:40.191315 20215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "6903cbdd40d548e99ac2d22fba404cb8"
format_stamp: "Formatted at 2026-06-20 18:44:40 on dist-test-slave-zfp8"
server_key: "871fc65df925f7cb678894e845e9179c"
server_key_iv: "9d3ab1dced12f9fddfd90171b3708761"
server_key_version: "encryptionkey@0"
I20260620 18:44:40.191588 20215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:40.209892 20215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:40.214427 20215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:40.214720 20215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:40.215792 20215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:40.217209 20215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:40.217334 20215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:40.217463 20215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:40.217549 20215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:40.250272 20215 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:38455
I20260620 18:44:40.250358 20341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:38455 every 8 connection(s)
I20260620 18:44:40.251178 20215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:40.255146 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 20215
I20260620 18:44:40.255402 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:40.255821 15771 external_mini_cluster.cc:1428] Setting key ad35ec77d30fdde14da2bec26fc33db6
Jun 20 18:44:40 dist-test-slave-zfp8 krb5kdc[19845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981080, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:40.270140 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.253639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:44815 (local address 127.15.102.254:46863)
0620 18:44:40.253819 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.253824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.254994 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 18:44:40.256939 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.256945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.257038 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.257144 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:40.258601 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.259416 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.260928 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.261336 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.261468 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.264789 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:40.264794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:40.264797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:40.264829 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:40.267192 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.267929 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.267936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.267942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.268072 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.268478 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.268484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.268487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.268741 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:40.268902 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.269570 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.269774 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:44:40.271184 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.254156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46863 (local address 127.15.102.195:44815)
0620 18:44:40.254833 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.254871 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:40.256674 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:40.257382 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:40.257390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:40.257880 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:40.258430 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.258447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.259564 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.259569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.260770 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.260781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.261559 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.261566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.261788 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.262386 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:40.262412 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:40.264572 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:40.267385 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:40.267394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:40.267417 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:40.267809 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:40.268180 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:40.268185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:40.268189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:40.268368 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:40.268881 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:40.268888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:40.269207 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 18:44:40.269486 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 18:44:40.269978 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":257,"threads_started":1}
I20260620 18:44:40.272563 20342 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46863
I20260620 18:44:40.272810 20342 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:40.273392 20342 heartbeater.cc:511] Master 127.15.102.254:46863 requested a full tablet report, sending...
I20260620 18:44:40.274637 19889 ts_manager.cc:194] Registered new tserver with Master: 6903cbdd40d548e99ac2d22fba404cb8 (127.15.102.195:38455)
I20260620 18:44:40.275525 19889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:44815
I20260620 18:44:40.282811 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:44:40.300328 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.287099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.15.102.254:46863)
0620 18:44:40.287318 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.287327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.287799 (+ 472us) server_negotiation.cc:407] Connection header received
0620 18:44:40.287927 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.287931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.287989 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.288068 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:40.289573 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.290511 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.292178 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.292718 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.292853 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.295106 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:40.295112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:40.295117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:40.295157 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:40.297949 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.298524 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.298528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.298531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.298680 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.298995 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.298998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.299000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.299373 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:40.299549 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.299786 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.299943 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:44:40.304278 19889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:44:40.310417 20077 heartbeater.cc:503] Master 127.15.102.254:46863 was elected leader, sending a full tablet report...
W20260620 18:44:40.311354 19889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:40.346622 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.323626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.15.102.194:40485)
0620 18:44:40.336629 (+ 13003us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.336640 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.336670 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:44:40.336775 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.336798 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.337064 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.337290 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:40.339028 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.340499 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.343614 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.345118 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.345300 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.345479 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.346129 (+ 650us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.346306 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12845,"thread_start_us":12709,"threads_started":1}
I20260620 18:44:40.346691 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.336482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34897 (local address 127.0.0.1:44238)
0620 18:44:40.338162 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.338185 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:40.338288 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:40.339492 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:40.339498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:40.339524 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:40.339879 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.339889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.341515 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.341523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.343476 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.343489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.345401 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.345406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.345778 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.345912 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 18:44:40.346044 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 18:44:40.346235 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1535,"mutex_wait_us":212,"thread_start_us":1381,"threads_started":1}
I20260620 18:44:40.347544 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.323384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:40485 (local address 127.0.0.1:54340)
0620 18:44:40.335815 (+ 12431us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.335882 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:40.336093 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:40.337617 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:40.337624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:40.337673 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:40.338837 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.338850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.340645 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.340651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.343458 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.343466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.345391 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.345398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.345774 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.345912 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 18:44:40.346954 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0620 18:44:40.347089 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12090,"mutex_wait_us":906,"spinlock_wait_cycles":2048,"thread_start_us":11921,"threads_started":1}
I20260620 18:44:40.347635 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.336834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.15.102.193:34897)
0620 18:44:40.338853 (+ 2019us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.338861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.338890 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:44:40.339007 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.339016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.339285 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.339490 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:40.340042 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.341301 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.343671 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.345122 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.345328 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.345479 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.347109 (+ 1630us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.347316 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1840,"thread_start_us":1665,"threads_started":1}
I20260620 18:44:40.348153 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.323468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:38455 (local address 127.0.0.1:42248)
0620 18:44:40.335815 (+ 12347us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.336588 (+ 773us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:40.336790 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:40.337617 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:40.337624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:40.337675 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:40.338208 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.338225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.340735 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.340745 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.343446 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.343465 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.347414 (+ 3949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.347422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.347583 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.347633 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:44:40.347739 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:44:40.347858 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12223,"mutex_wait_us":117,"spinlock_wait_cycles":8064,"thread_start_us":12345,"threads_started":1}
I20260620 18:44:40.348830 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.323970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.15.102.195:38455)
0620 18:44:40.334829 (+ 10859us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.334843 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.336688 (+ 1845us) server_negotiation.cc:407] Connection header received
0620 18:44:40.336965 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.336971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.337144 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.337301 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:40.339071 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.340625 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.345567 (+ 4942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.347076 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.347315 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.347492 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.348399 (+ 907us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.348509 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10581,"thread_start_us":10883,"threads_started":1}
I20260620 18:44:40.351171 20145 tablet_service.cc:1467] Processing CreateTablet for tablet 68e05475425e47fab2523cd5a92d9693 (DEFAULT_TABLE table=test-table [id=4e05bb91405d4ba29fd6f57b84111394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:40.351187 20012 tablet_service.cc:1467] Processing CreateTablet for tablet 68e05475425e47fab2523cd5a92d9693 (DEFAULT_TABLE table=test-table [id=4e05bb91405d4ba29fd6f57b84111394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:40.352847 20012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e05475425e47fab2523cd5a92d9693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:40.352864 20145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e05475425e47fab2523cd5a92d9693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:40.355154 20277 tablet_service.cc:1467] Processing CreateTablet for tablet 68e05475425e47fab2523cd5a92d9693 (DEFAULT_TABLE table=test-table [id=4e05bb91405d4ba29fd6f57b84111394]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:40.356518 20277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68e05475425e47fab2523cd5a92d9693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:40.364545 20360 tablet_bootstrap.cc:492] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Bootstrap starting.
I20260620 18:44:40.368010 20361 tablet_bootstrap.cc:492] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Bootstrap starting.
I20260620 18:44:40.368095 20360 tablet_bootstrap.cc:654] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:40.370110 20360 log.cc:824] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:40.371032 20362 tablet_bootstrap.cc:492] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Bootstrap starting.
I20260620 18:44:40.372303 20361 tablet_bootstrap.cc:654] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:40.373663 20361 log.cc:824] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:40.374104 20362 tablet_bootstrap.cc:654] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:40.374819 20360 tablet_bootstrap.cc:492] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: No bootstrap required, opened a new log
I20260620 18:44:40.375028 20362 log.cc:824] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:40.375182 20360 ts_tablet_manager.cc:1397] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260620 18:44:40.377784 20361 tablet_bootstrap.cc:492] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: No bootstrap required, opened a new log
I20260620 18:44:40.378046 20361 ts_tablet_manager.cc:1397] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:44:40.378103 20362 tablet_bootstrap.cc:492] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: No bootstrap required, opened a new log
I20260620 18:44:40.378429 20362 ts_tablet_manager.cc:1397] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 18:44:40.383594 20360 raft_consensus.cc:348] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
I20260620 18:44:40.384013 20360 raft_consensus.cc:374] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:40.384164 20360 raft_consensus.cc:729] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d18460526c641ae9054efae46b3bae6, State: Initialized, Role: FOLLOWER
I20260620 18:44:40.384881 20360 consensus_queue.cc:260] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
I20260620 18:44:40.387120 20360 ts_tablet_manager.cc:1428] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260620 18:44:40.388618 20361 raft_consensus.cc:348] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
I20260620 18:44:40.388698 20362 raft_consensus.cc:348] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
I20260620 18:44:40.389074 20361 raft_consensus.cc:374] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:40.389074 20362 raft_consensus.cc:374] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:40.389173 20361 raft_consensus.cc:729] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be73fd67e364d7ba9cbca065fb3a581, State: Initialized, Role: FOLLOWER
I20260620 18:44:40.389179 20362 raft_consensus.cc:729] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6903cbdd40d548e99ac2d22fba404cb8, State: Initialized, Role: FOLLOWER
I20260620 18:44:40.389770 20361 consensus_queue.cc:260] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
I20260620 18:44:40.389768 20362 consensus_queue.cc:260] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
I20260620 18:44:40.391726 20361 ts_tablet_manager.cc:1428] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 18:44:40.392088 20342 heartbeater.cc:503] Master 127.15.102.254:46863 was elected leader, sending a full tablet report...
I20260620 18:44:40.392213 20210 heartbeater.cc:503] Master 127.15.102.254:46863 was elected leader, sending a full tablet report...
I20260620 18:44:40.392601 20362 ts_tablet_manager.cc:1428] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260620 18:44:40.477563 20366 raft_consensus.cc:482] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:40.477965 20366 raft_consensus.cc:504] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
I20260620 18:44:40.480614 20366 leader_election.cc:288] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2be73fd67e364d7ba9cbca065fb3a581 (127.15.102.194:40485), 6903cbdd40d548e99ac2d22fba404cb8 (127.15.102.195:38455)
I20260620 18:44:40.488262 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.481553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:59319 (local address 127.15.102.195:38455)
0620 18:44:40.481755 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.481764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.481790 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:44:40.482104 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.482109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.482170 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.482287 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:40.482889 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.484162 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.486503 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.487339 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.487523 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.487603 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.487884 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.488004 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:44:40.488343 20369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.480971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:38455 (local address 127.15.102.193:59319)
0620 18:44:40.481683 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.481712 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:40.481894 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:40.482300 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:40.482305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:40.482330 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:40.482731 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.482749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.484339 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.484344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.486339 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.486351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.487558 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.487565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.487718 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.487741 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:44:40.487848 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 18:44:40.487978 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":828,"threads_started":1}
I20260620 18:44:40.489241 20297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e05475425e47fab2523cd5a92d9693" candidate_uuid: "4d18460526c641ae9054efae46b3bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6903cbdd40d548e99ac2d22fba404cb8" is_pre_election: true
I20260620 18:44:40.489779 20297 raft_consensus.cc:2454] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d18460526c641ae9054efae46b3bae6 in term 0.
I20260620 18:44:40.490482 19967 leader_election.cc:302] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d18460526c641ae9054efae46b3bae6, 6903cbdd40d548e99ac2d22fba404cb8; no voters:
I20260620 18:44:40.490676 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.481403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:51973 (local address 127.15.102.194:40485)
0620 18:44:40.481618 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.481625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.481813 (+ 188us) server_negotiation.cc:407] Connection header received
0620 18:44:40.482104 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.482113 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.482205 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.482267 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:40.483179 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.484410 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.487682 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.489128 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.489290 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.489360 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.490135 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.490386 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:44:40.490718 20370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.481037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:40485 (local address 127.15.102.193:51973)
0620 18:44:40.481707 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.481735 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:40.481924 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:40.482525 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:40.482531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:40.482562 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:40.482979 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.482996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.484564 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.484571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.487449 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:40.487467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.489729 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.489736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:40.489940 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.489988 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:44:40.490091 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:44:40.490275 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":593,"threads_started":1}
I20260620 18:44:40.490922 20366 raft_consensus.cc:2781] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:40.491067 20366 raft_consensus.cc:482] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:40.491178 20366 raft_consensus.cc:3037] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:40.492070 20165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e05475425e47fab2523cd5a92d9693" candidate_uuid: "4d18460526c641ae9054efae46b3bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be73fd67e364d7ba9cbca065fb3a581" is_pre_election: true
I20260620 18:44:40.492599 20165 raft_consensus.cc:2454] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d18460526c641ae9054efae46b3bae6 in term 0.
I20260620 18:44:40.494083 20366 raft_consensus.cc:504] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
I20260620 18:44:40.494684 20366 leader_election.cc:288] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [CANDIDATE]: Term 1 election: Requested vote from peers 2be73fd67e364d7ba9cbca065fb3a581 (127.15.102.194:40485), 6903cbdd40d548e99ac2d22fba404cb8 (127.15.102.195:38455)
I20260620 18:44:40.495133 20165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e05475425e47fab2523cd5a92d9693" candidate_uuid: "4d18460526c641ae9054efae46b3bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2be73fd67e364d7ba9cbca065fb3a581"
I20260620 18:44:40.495338 20165 raft_consensus.cc:3037] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:40.495223 20297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68e05475425e47fab2523cd5a92d9693" candidate_uuid: "4d18460526c641ae9054efae46b3bae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6903cbdd40d548e99ac2d22fba404cb8"
I20260620 18:44:40.495515 20297 raft_consensus.cc:3037] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:40.497659 20165 raft_consensus.cc:2454] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d18460526c641ae9054efae46b3bae6 in term 1.
I20260620 18:44:40.497829 20297 raft_consensus.cc:2454] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d18460526c641ae9054efae46b3bae6 in term 1.
I20260620 18:44:40.498064 19966 leader_election.cc:302] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2be73fd67e364d7ba9cbca065fb3a581, 4d18460526c641ae9054efae46b3bae6; no voters:
I20260620 18:44:40.498389 20366 raft_consensus.cc:2781] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:40.498797 20366 raft_consensus.cc:686] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 1 LEADER]: Becoming Leader. State: Replica: 4d18460526c641ae9054efae46b3bae6, State: Running, Role: LEADER
I20260620 18:44:40.499372 20366 consensus_queue.cc:237] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } }
W20260620 18:44:40.502713 20343 tablet.cc:2369] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:40.503901 19888 catalog_manager.cc:5495] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d18460526c641ae9054efae46b3bae6 (127.15.102.193). New cstate: current_term: 1 leader_uuid: "4d18460526c641ae9054efae46b3bae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d18460526c641ae9054efae46b3bae6" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34897 } health_report { overall_health: HEALTHY } } }
W20260620 18:44:40.515727 20211 tablet.cc:2369] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:44:40.531517 20078 tablet.cc:2369] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:40.541379 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.536301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.15.102.193:34897)
0620 18:44:40.536549 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.536553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.536571 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:44:40.536756 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.536760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.536846 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.536976 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:40.537287 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.537951 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.539444 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.539825 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.539918 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.540331 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:40.540819 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:40.540929 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.541016 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.541076 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 18:44:40.549141 20165 raft_consensus.cc:1261] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 1 FOLLOWER]: Refusing update from remote peer 4d18460526c641ae9054efae46b3bae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:40.549182 20297 raft_consensus.cc:1261] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 1 FOLLOWER]: Refusing update from remote peer 4d18460526c641ae9054efae46b3bae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:40.550135 20371 consensus_queue.cc:1035] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6903cbdd40d548e99ac2d22fba404cb8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:40.550458 20366 consensus_queue.cc:1035] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be73fd67e364d7ba9cbca065fb3a581" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:40.575305 19888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:40.575587 19888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:40.578879 19888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2acaedc505a4168b86181b60606d547: Sending DeleteTablet for 3 replicas of tablet 68e05475425e47fab2523cd5a92d9693
I20260620 18:44:40.580085 20145 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e05475425e47fab2523cd5a92d9693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:40 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:54340
I20260620 18:44:40.580483 20277 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e05475425e47fab2523cd5a92d9693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:40 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:42248
I20260620 18:44:40.580492 20012 tablet_service.cc:1514] Processing DeleteTablet for tablet 68e05475425e47fab2523cd5a92d9693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:40 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:44238
I20260620 18:44:40.581413 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19950
I20260620 18:44:40.592554 20383 tablet_replica.cc:330] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: stopping tablet replica
I20260620 18:44:40.593113 20385 tablet_replica.cc:330] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: stopping tablet replica
I20260620 18:44:40.593770 20385 raft_consensus.cc:2229] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:40.594305 20383 raft_consensus.cc:2229] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:40.594913 20385 raft_consensus.cc:2258] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:40.595435 20383 raft_consensus.cc:2258] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:40.596607 20384 tablet_replica.cc:330] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: stopping tablet replica
I20260620 18:44:40.598119 20383 ts_tablet_manager.cc:1905] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:40.598146 20385 ts_tablet_manager.cc:1905] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:40.604362 20384 raft_consensus.cc:2229] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:40.605181 20384 raft_consensus.cc:2258] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:40.605572 20385 ts_tablet_manager.cc:1918] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 20 18:44:40 dist-test-slave-zfp8 krb5kdc[19845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:40.605723 20385 log.cc:1196] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/68e05475425e47fab2523cd5a92d9693
I20260620 18:44:40.606338 20385 ts_tablet_manager.cc:1939] T 68e05475425e47fab2523cd5a92d9693 P 6903cbdd40d548e99ac2d22fba404cb8: Deleting consensus metadata
I20260620 18:44:40.607612 20384 ts_tablet_manager.cc:1905] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:40.608335 19875 catalog_manager.cc:4841] TS 6903cbdd40d548e99ac2d22fba404cb8 (127.15.102.195:38455): tablet 68e05475425e47fab2523cd5a92d9693 (table test-table [id=4e05bb91405d4ba29fd6f57b84111394]) successfully deleted
I20260620 18:44:40.609838 20383 ts_tablet_manager.cc:1918] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:40.610055 20383 log.cc:1196] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/68e05475425e47fab2523cd5a92d9693
I20260620 18:44:40.610590 20383 ts_tablet_manager.cc:1939] T 68e05475425e47fab2523cd5a92d9693 P 2be73fd67e364d7ba9cbca065fb3a581: Deleting consensus metadata
I20260620 18:44:40.612478 19874 catalog_manager.cc:4841] TS 2be73fd67e364d7ba9cbca065fb3a581 (127.15.102.194:40485): tablet 68e05475425e47fab2523cd5a92d9693 (table test-table [id=4e05bb91405d4ba29fd6f57b84111394]) successfully deleted
I20260620 18:44:40.615401 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.584041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.15.102.193:34897)
0620 18:44:40.584658 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.584667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.593921 (+ 9254us) server_negotiation.cc:407] Connection header received
0620 18:44:40.594536 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.594546 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.594668 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.594760 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:40.596890 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.598160 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.600071 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.600788 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.600929 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.606979 (+ 6050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:40.606992 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:40.607010 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:40.607050 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:40.611915 (+ 4865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.612844 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.612850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.612852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.613009 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.613434 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.613439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.613444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.613790 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:40.614135 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.614598 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.614883 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452}
I20260620 18:44:40.615744 20384 ts_tablet_manager.cc:1918] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:40.615901 20384 log.cc:1196] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/68e05475425e47fab2523cd5a92d9693
I20260620 18:44:40.616252 20072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:40.719134 20384 ts_tablet_manager.cc:1939] T 68e05475425e47fab2523cd5a92d9693 P 4d18460526c641ae9054efae46b3bae6: Deleting consensus metadata
I20260620 18:44:40.719182 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 19950
I20260620 18:44:40.737306 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 20083
W20260620 18:44:40.737285 19875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:34897 (error 108)
I20260620 18:44:40.739100 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.738695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34897 (local address 127.0.0.1:60376)
0620 18:44:40.738868 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.738960 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 18:44:40.739352 19875 catalog_manager.cc:4584] TS 4d18460526c641ae9054efae46b3bae6 (127.15.102.193:34897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68e05475425e47fab2523cd5a92d9693: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34897: connect: Connection refused (error 111)
Jun 20 18:44:40 dist-test-slave-zfp8 krb5kdc[19845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:44:40.753144 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.738175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.15.102.194:40485)
0620 18:44:40.738383 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.738390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.738499 (+ 109us) server_negotiation.cc:407] Connection header received
0620 18:44:40.738750 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.738763 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.738833 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.738944 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:40.740404 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.741202 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.742505 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.742913 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.743063 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.747170 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:40.747184 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:40.747200 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:40.747294 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:40.750162 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.750925 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.750933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.750935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.751067 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.751517 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.751524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.751528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.751904 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:40.752243 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.752553 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.752789 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:44:40.753849 20205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:40.793069 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.792539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34897 (local address 127.0.0.1:60390)
0620 18:44:40.792755 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.792883 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:44:40.794073 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.793697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34897 (local address 127.0.0.1:60406)
0620 18:44:40.793847 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.793942 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 18:44:40.841969 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 20083
I20260620 18:44:40.854774 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.854276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34897 (local address 127.0.0.1:60420)
0620 18:44:40.854551 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.854639 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 18:44:40.855842 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.855506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34897 (local address 127.0.0.1:60426)
0620 18:44:40.855650 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.855724 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:44:40.859395 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 20215
Jun 20 18:44:40 dist-test-slave-zfp8 krb5kdc[19845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:44:40.873587 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.860142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.15.102.195:38455)
0620 18:44:40.860311 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:40.860316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:40.860337 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:44:40.860399 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:40.860403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:40.860478 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:40.860575 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:40.861770 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.862576 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.863945 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:40.864355 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:40.864459 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:40.868587 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:40.868601 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:40.868616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:40.868649 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:40.871464 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.872013 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.872019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.872022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.872146 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:40.872445 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:40.872448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:40.872450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:40.872773 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:40.872990 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:40.873188 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 18:44:40.873351 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:44:40.874224 20337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:40.936564 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.936004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34897 (local address 127.0.0.1:60430)
0620 18:44:40.936305 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.936398 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260620 18:44:40.937561 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:40.937179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34897 (local address 127.0.0.1:60440)
0620 18:44:40.937385 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:40.937450 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:44:40.962410 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 20215
I20260620 18:44:40.978574 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 19858
I20260620 18:44:40.979839 19919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:41.086709 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 19858
2026-06-20T18:44:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4136 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T18:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:41Z Disabled control of system clock
I20260620 18:44:41.152592 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:38127
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:38127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:44:41.468998 20395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:41.469305 20395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:41.469399 20395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:41.476549 20395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:41.476667 20395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:41.476704 20395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:41.476752 20395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:41.476796 20395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:41.485903 20395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:38127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:41.487171 20395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:41.489038 20395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:41.500666 20400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:41.500748 20401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:41.500699 20403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:41.501955 20395 server_base.cc:1034] running on GCE node
I20260620 18:44:41.502745 20395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:41.504146 20395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:41.505353 20395 hybrid_clock.cc:648] HybridClock initialized: now 1781981081505318 us; error 29 us; skew 500 ppm
I20260620 18:44:41.505648 20395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:44:41.507809 20395 webserver.cc:460] Webserver started at http://127.15.102.254:34483/ using document root <none> and password file <none>
I20260620 18:44:41.508483 20395 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:41.508570 20395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:41.508917 20395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:41.511871 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "0daa83527ec34d29b99531bc4204fe62"
format_stamp: "Formatted at 2026-06-20 18:44:41 on dist-test-slave-zfp8"
server_key: "128c0578bdd5e1e1e3c2710a4f13d7a1"
server_key_iv: "c899a8f5e869fa23a42d06a90cadb403"
server_key_version: "encryptionkey@0"
I20260620 18:44:41.512676 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "0daa83527ec34d29b99531bc4204fe62"
format_stamp: "Formatted at 2026-06-20 18:44:41 on dist-test-slave-zfp8"
server_key: "128c0578bdd5e1e1e3c2710a4f13d7a1"
server_key_iv: "c899a8f5e869fa23a42d06a90cadb403"
server_key_version: "encryptionkey@0"
I20260620 18:44:41.517716 20395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:44:41.521130 20409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:41.522430 20395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:41.522605 20395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "0daa83527ec34d29b99531bc4204fe62"
format_stamp: "Formatted at 2026-06-20 18:44:41 on dist-test-slave-zfp8"
server_key: "128c0578bdd5e1e1e3c2710a4f13d7a1"
server_key_iv: "c899a8f5e869fa23a42d06a90cadb403"
server_key_version: "encryptionkey@0"
I20260620 18:44:41.522820 20395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:41.528494 20395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:41.529285 20395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:41.529595 20395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:41.547560 20395 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:38127
I20260620 18:44:41.547592 20460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:38127 every 8 connection(s)
I20260620 18:44:41.548919 20395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:41.549564 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 20395
I20260620 18:44:41.549789 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:41.550535 15771 external_mini_cluster.cc:1428] Setting key 38a62f5297ffcbcbc9e85b206539fd8b
I20260620 18:44:41.553406 20461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:41.560159 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:41.552664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.15.102.254:38127)
0620 18:44:41.553713 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:41.553727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:41.553779 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:44:41.556027 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:41.556054 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:41.556623 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:41.557064 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:41.558081 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:41.558090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:41.558107 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:41.558123 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:41.558210 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:41.558237 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:41.558479 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:41.558603 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 18:44:41.558876 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":319,"threads_started":1}
I20260620 18:44:41.566061 20461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62: Bootstrap starting.
I20260620 18:44:41.569581 20461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:41.570513 20461 log.cc:824] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:41.573153 20461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62: No bootstrap required, opened a new log
I20260620 18:44:41.580250 20461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daa83527ec34d29b99531bc4204fe62" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38127 } }
I20260620 18:44:41.580603 20461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:41.580675 20461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daa83527ec34d29b99531bc4204fe62, State: Initialized, Role: FOLLOWER
I20260620 18:44:41.581243 20461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daa83527ec34d29b99531bc4204fe62" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38127 } }
I20260620 18:44:41.581449 20461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:41.581543 20461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:41.581663 20461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:41.584241 20461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daa83527ec34d29b99531bc4204fe62" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38127 } }
I20260620 18:44:41.584714 20461 leader_election.cc:302] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0daa83527ec34d29b99531bc4204fe62; no voters:
I20260620 18:44:41.585248 20461 leader_election.cc:288] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:41.585420 20466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:41.586162 20466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [term 1 LEADER]: Becoming Leader. State: Replica: 0daa83527ec34d29b99531bc4204fe62, State: Running, Role: LEADER
I20260620 18:44:41.586737 20466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daa83527ec34d29b99531bc4204fe62" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38127 } }
I20260620 18:44:41.587466 20461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:41.589599 20467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0daa83527ec34d29b99531bc4204fe62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daa83527ec34d29b99531bc4204fe62" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38127 } } }
I20260620 18:44:41.589699 20468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0daa83527ec34d29b99531bc4204fe62. Latest consensus state: current_term: 1 leader_uuid: "0daa83527ec34d29b99531bc4204fe62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daa83527ec34d29b99531bc4204fe62" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38127 } } }
I20260620 18:44:41.589982 20467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:41.590005 20468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:41.596215 20474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:41.601645 20474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:41.611486 20474 catalog_manager.cc:1347] Generated new cluster ID: e2e09ce4e98e410192486c6f015edd52
I20260620 18:44:41.611606 20474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:41.622411 20474 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:41.622875 20474 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:41.631467 20474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62: Generated new TSK 0
I20260620 18:44:41.632494 20474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:44:41.645748 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38127
--builtin_ntp_servers=127.15.102.212:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 18:44:41.972211 20485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:41.972573 20485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:41.972666 20485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:41.979676 20485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:41.979770 20485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:41.979875 20485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:41.988931 20485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:41.990159 20485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:41.991853 20485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:42.007634 20490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:42.007674 20491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:42.007632 20493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:42.008337 20485 server_base.cc:1034] running on GCE node
I20260620 18:44:42.009001 20485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:42.010002 20485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:42.011296 20485 hybrid_clock.cc:648] HybridClock initialized: now 1781981082011194 us; error 69 us; skew 500 ppm
I20260620 18:44:42.011647 20485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:44:42.013906 20485 webserver.cc:460] Webserver started at http://127.15.102.193:35161/ using document root <none> and password file <none>
I20260620 18:44:42.014621 20485 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:42.014756 20485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:42.015098 20485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:42.018136 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "6bee0c40ddc4445da142f3ada67e43a1"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "1e3dbfc4d0b99f5bfcee0753bbbc22aa"
server_key_iv: "ce4bcfad3654827b7d27dc6aef7af559"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.018946 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "6bee0c40ddc4445da142f3ada67e43a1"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "1e3dbfc4d0b99f5bfcee0753bbbc22aa"
server_key_iv: "ce4bcfad3654827b7d27dc6aef7af559"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.024150 20485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:44:42.027666 20499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.029111 20485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:44:42.029310 20485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "6bee0c40ddc4445da142f3ada67e43a1"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "1e3dbfc4d0b99f5bfcee0753bbbc22aa"
server_key_iv: "ce4bcfad3654827b7d27dc6aef7af559"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.029534 20485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:42.035050 20485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:42.035948 20485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:42.036218 20485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:42.037087 20485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:42.038456 20485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:42.038576 20485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.038703 20485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:42.038789 20485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.071411 20485 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:42775
I20260620 18:44:42.071453 20611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:42775 every 8 connection(s)
I20260620 18:44:42.072722 20485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:42.074496 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 20485
I20260620 18:44:42.074682 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:42.075405 15771 external_mini_cluster.cc:1428] Setting key 341795eefa93b571d6c42d7991960880
I20260620 18:44:42.078122 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38127
--builtin_ntp_servers=127.15.102.212:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:44:42.081775 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.075634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:51879 (local address 127.15.102.254:38127)
0620 18:44:42.076136 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:42.076144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:42.077620 (+ 1476us) server_negotiation.cc:407] Connection header received
0620 18:44:42.079409 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:42.079418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:42.079485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:42.079604 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:42.080940 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:42.080948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:42.080962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:42.080985 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:42.081039 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:42.081058 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:42.081180 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:42.081344 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 18:44:42.081421 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
I20260620 18:44:42.082592 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.076334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:38127 (local address 127.15.102.193:51879)
0620 18:44:42.077452 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:42.077497 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:42.079085 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:42.079797 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:42.079805 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:42.079900 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:42.080628 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:44:42.080630 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:42.080674 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:44:42.080757 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:42.081155 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:42.081161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:42.081167 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:44:42.081400 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 18:44:42.081598 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":351,"threads_started":1}
I20260620 18:44:42.084112 20612 heartbeater.cc:347] Connected to a master server at 127.15.102.254:38127
I20260620 18:44:42.084437 20612 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:42.085011 20612 heartbeater.cc:511] Master 127.15.102.254:38127 requested a full tablet report, sending...
I20260620 18:44:42.087169 20426 ts_manager.cc:194] Registered new tserver with Master: 6bee0c40ddc4445da142f3ada67e43a1 (127.15.102.193:42775)
W20260620 18:44:42.401037 20616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:42.401297 20616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:42.401387 20616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:42.408748 20616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:42.408843 20616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:42.408949 20616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:42.417922 20616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:42.419162 20616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:42.420944 20616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:42.433858 20622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:42.434072 20621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:42.434195 20624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:42.434990 20616 server_base.cc:1034] running on GCE node
I20260620 18:44:42.435688 20616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:42.436607 20616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:42.437829 20616 hybrid_clock.cc:648] HybridClock initialized: now 1781981082437790 us; error 51 us; skew 500 ppm
I20260620 18:44:42.438138 20616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:44:42.440310 20616 webserver.cc:460] Webserver started at http://127.15.102.194:36555/ using document root <none> and password file <none>
I20260620 18:44:42.440980 20616 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:42.441111 20616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:42.441437 20616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:42.444247 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "10558c01a86f0a9635e44cdb50f33bf4"
server_key_iv: "f6873e2abeba4ea123e79113930d5cab"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.445024 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "10558c01a86f0a9635e44cdb50f33bf4"
server_key_iv: "f6873e2abeba4ea123e79113930d5cab"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.449893 20616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:44:42.454581 20630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.456168 20616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:44:42.456364 20616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "10558c01a86f0a9635e44cdb50f33bf4"
server_key_iv: "f6873e2abeba4ea123e79113930d5cab"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.456707 20616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:42.464125 20616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:42.464911 20616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:42.465137 20616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:42.465955 20616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:42.467211 20616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:42.467383 20616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.467473 20616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:42.467530 20616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.500025 20616 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:33537
I20260620 18:44:42.500098 20742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:33537 every 8 connection(s)
I20260620 18:44:42.501499 20616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:42.503336 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 20616
I20260620 18:44:42.503533 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:42.504108 15771 external_mini_cluster.cc:1428] Setting key 3a7fa62b824520bc1fce66f17ad911de
I20260620 18:44:42.506778 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38127
--builtin_ntp_servers=127.15.102.212:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 18:44:42.510223 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.505331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:45835 (local address 127.15.102.254:38127)
0620 18:44:42.505520 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:42.505526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:42.506082 (+ 556us) server_negotiation.cc:407] Connection header received
0620 18:44:42.507938 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:42.507944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:42.508019 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:42.508130 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:42.509427 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:42.509435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:42.509450 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:42.509472 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:42.509511 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:42.509531 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:42.509653 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:42.509814 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 18:44:42.509877 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:44:42.511091 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.504911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:38127 (local address 127.15.102.194:45835)
0620 18:44:42.505942 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:42.505987 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:42.507679 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:42.508320 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:42.508327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:42.508444 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:42.509148 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:44:42.509150 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:42.509188 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:44:42.509269 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:42.509629 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:42.509636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:42.509641 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:44:42.509844 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 18:44:42.510040 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"spinlock_wait_cycles":19328,"thread_start_us":398,"threads_started":1}
I20260620 18:44:42.512530 20743 heartbeater.cc:347] Connected to a master server at 127.15.102.254:38127
I20260620 18:44:42.512847 20743 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:42.513412 20743 heartbeater.cc:511] Master 127.15.102.254:38127 requested a full tablet report, sending...
I20260620 18:44:42.514755 20426 ts_manager.cc:194] Registered new tserver with Master: b7b4ec2ce5aa4b93b459289ac2bc9b1a (127.15.102.194:33537)
W20260620 18:44:42.841747 20746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:42.842075 20746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:42.842233 20746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:42.849846 20746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:42.849988 20746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:42.850183 20746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:42.859483 20746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:38127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:42.860853 20746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:42.862753 20746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:42.875655 20752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:42.875722 20754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:42.875736 20751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:42.876355 20746 server_base.cc:1034] running on GCE node
I20260620 18:44:42.876924 20746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:42.877830 20746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:42.879066 20746 hybrid_clock.cc:648] HybridClock initialized: now 1781981082879017 us; error 40 us; skew 500 ppm
I20260620 18:44:42.879422 20746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:44:42.881523 20746 webserver.cc:460] Webserver started at http://127.15.102.195:40499/ using document root <none> and password file <none>
I20260620 18:44:42.882191 20746 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:42.882318 20746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:42.882643 20746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:42.885427 20746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "9fd2f8b22ab94b578855028d186b513d"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "6f2756e4bc67d72c616889a1c3fb23e2"
server_key_iv: "8befc6e5a8bf6d5925cbd6368db3f92f"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.886188 20746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "9fd2f8b22ab94b578855028d186b513d"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "6f2756e4bc67d72c616889a1c3fb23e2"
server_key_iv: "8befc6e5a8bf6d5925cbd6368db3f92f"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.891079 20746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 18:44:42.894424 20760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.895731 20746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:42.895908 20746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "9fd2f8b22ab94b578855028d186b513d"
format_stamp: "Formatted at 2026-06-20 18:44:42 on dist-test-slave-zfp8"
server_key: "6f2756e4bc67d72c616889a1c3fb23e2"
server_key_iv: "8befc6e5a8bf6d5925cbd6368db3f92f"
server_key_version: "encryptionkey@0"
I20260620 18:44:42.896121 20746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:42.901865 20746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:42.902632 20746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:42.902904 20746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:42.903836 20746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:42.905140 20746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:42.905260 20746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.905390 20746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:42.905476 20746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:42.937091 20746 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:34185
I20260620 18:44:42.937171 20872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:34185 every 8 connection(s)
I20260620 18:44:42.938373 20746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:42.946434 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.940825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:59419 (local address 127.15.102.254:38127)
0620 18:44:42.940991 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:42.940996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:42.942377 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 18:44:42.944288 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:42.944294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:42.944370 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:42.944478 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:42.945684 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:42.945690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:42.945694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:42.945712 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:42.945749 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:42.945761 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:42.945907 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:42.946097 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 18:44:42.946165 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:44:42.947359 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.941330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:38127 (local address 127.15.102.195:59419)
0620 18:44:42.942191 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:42.942226 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:42.944023 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:42.944703 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:42.944711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:42.944803 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:42.945404 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:44:42.945407 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:42.945442 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:44:42.945518 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:42.945870 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:42.945881 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:42.945891 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:44:42.946166 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 18:44:42.946415 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":254,"threads_started":1}
I20260620 18:44:42.948469 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 20746
I20260620 18:44:42.948628 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:42.948861 20873 heartbeater.cc:347] Connected to a master server at 127.15.102.254:38127
I20260620 18:44:42.949146 15771 external_mini_cluster.cc:1428] Setting key 450d7cce964dfd064b42a38be9d109c8
I20260620 18:44:42.949151 20873 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:42.949683 20873 heartbeater.cc:511] Master 127.15.102.254:38127 requested a full tablet report, sending...
I20260620 18:44:42.951169 20426 ts_manager.cc:194] Registered new tserver with Master: 9fd2f8b22ab94b578855028d186b513d (127.15.102.195:34185)
I20260620 18:44:42.952123 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:44:42.958922 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.956562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.15.102.254:38127)
0620 18:44:42.956712 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:42.956716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:42.957228 (+ 512us) server_negotiation.cc:407] Connection header received
0620 18:44:42.957347 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:42.957351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:42.957402 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:42.957482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:42.958414 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:42.958418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:42.958421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:42.958436 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:42.958465 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:42.958476 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:42.958590 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:42.958671 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:44:42.958713 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:44:42.963009 20425 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:42.969529 20425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:43.002606 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.981867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.15.102.195:34185)
0620 18:44:42.996390 (+ 14523us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:42.996404 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:42.997647 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 18:44:42.997738 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:42.997751 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:42.998079 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:42.998259 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.001568 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.001577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.001591 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.001611 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.001674 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.001694 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.001885 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.002063 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.002175 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14254,"thread_start_us":15392,"threads_started":1}
I20260620 18:44:43.003479 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.981602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:34185 (local address 127.0.0.1:35024)
0620 18:44:42.996982 (+ 15380us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:42.997008 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:42.997167 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:42.999827 (+ 2660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:42.999840 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:42.999963 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:43.000943 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:44:43.000946 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:43.000983 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:44:43.001044 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:43.001835 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:43.001851 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:43.001863 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:44:43.002093 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 18:44:43.002216 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15196,"thread_start_us":15018,"threads_started":1}
I20260620 18:44:43.004607 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.981156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.15.102.193:42775)
0620 18:44:43.001341 (+ 20185us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:43.001359 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:43.001391 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:44:43.001486 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:43.001493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:43.001718 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:43.002410 (+ 692us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.003810 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.003823 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.003833 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.003846 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.003884 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.003899 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.004077 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.004144 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.004232 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19597,"thread_start_us":19430,"threads_started":1}
I20260620 18:44:43.004642 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.980918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.0.0.1:57038)
0620 18:44:42.983064 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:42.983118 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:42.983288 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:43.001901 (+ 18613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:43.001909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:43.001930 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:43.002700 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:44:43.002706 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:43.002728 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:44:43.002777 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:43.004001 (+ 1224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:43.004006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:43.004011 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 18:44:43.004147 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 18:44:43.004198 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1800,"thread_start_us":1627,"threads_started":1}
I20260620 18:44:43.008538 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.983593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:33537 (local address 127.0.0.1:36660)
0620 18:44:43.004644 (+ 21051us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.004666 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:43.004903 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:43.006955 (+ 2052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:43.006958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:43.006987 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:43.007561 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:44:43.007564 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:43.007579 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:44:43.007619 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:43.008143 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:43.008147 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:43.008154 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:44:43.008252 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:44:43.008303 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20918,"thread_start_us":23172,"threads_started":1}
I20260620 18:44:43.009454 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:42.984173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.15.102.194:33537)
0620 18:44:43.005604 (+ 21431us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:43.005612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:43.005639 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:44:43.005700 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:43.005707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:43.005905 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:43.006045 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.007787 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.007796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.007808 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.007827 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.007892 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.007912 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.008087 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.009044 (+ 957us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.009139 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21276,"thread_start_us":21135,"threads_started":1}
I20260620 18:44:43.009339 20808 tablet_service.cc:1467] Processing CreateTablet for tablet a6261cae391d485fb38908b6f1da419c (DEFAULT_TABLE table=test-table [id=90e187bbab494199bf1cd9ce92df2304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:43.010000 20547 tablet_service.cc:1467] Processing CreateTablet for tablet a6261cae391d485fb38908b6f1da419c (DEFAULT_TABLE table=test-table [id=90e187bbab494199bf1cd9ce92df2304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:43.011335 20547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6261cae391d485fb38908b6f1da419c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:43.011833 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6261cae391d485fb38908b6f1da419c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:43.013305 20678 tablet_service.cc:1467] Processing CreateTablet for tablet a6261cae391d485fb38908b6f1da419c (DEFAULT_TABLE table=test-table [id=90e187bbab494199bf1cd9ce92df2304]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:43.014624 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6261cae391d485fb38908b6f1da419c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:43.022398 20891 tablet_bootstrap.cc:492] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1: Bootstrap starting.
I20260620 18:44:43.025318 20891 tablet_bootstrap.cc:654] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:43.026652 20892 tablet_bootstrap.cc:492] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: Bootstrap starting.
I20260620 18:44:43.026645 20891 log.cc:824] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:43.029413 20894 tablet_bootstrap.cc:492] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: Bootstrap starting.
I20260620 18:44:43.030975 20892 tablet_bootstrap.cc:654] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:43.032289 20892 log.cc:824] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:43.033001 20891 tablet_bootstrap.cc:492] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1: No bootstrap required, opened a new log
I20260620 18:44:43.033007 20894 tablet_bootstrap.cc:654] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:43.033267 20891 ts_tablet_manager.cc:1397] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260620 18:44:43.034274 20894 log.cc:824] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:43.035501 20892 tablet_bootstrap.cc:492] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: No bootstrap required, opened a new log
I20260620 18:44:43.035764 20892 ts_tablet_manager.cc:1397] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:44:43.036757 20894 tablet_bootstrap.cc:492] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: No bootstrap required, opened a new log
I20260620 18:44:43.037009 20894 ts_tablet_manager.cc:1397] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 18:44:43.043778 20891 raft_consensus.cc:348] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.044289 20891 raft_consensus.cc:374] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:43.044420 20891 raft_consensus.cc:729] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bee0c40ddc4445da142f3ada67e43a1, State: Initialized, Role: FOLLOWER
I20260620 18:44:43.044416 20894 raft_consensus.cc:348] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.044873 20894 raft_consensus.cc:374] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:43.045024 20894 raft_consensus.cc:729] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b4ec2ce5aa4b93b459289ac2bc9b1a, State: Initialized, Role: FOLLOWER
I20260620 18:44:43.045063 20891 consensus_queue.cc:260] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.045578 20894 consensus_queue.cc:260] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.045900 20892 raft_consensus.cc:348] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.046321 20612 heartbeater.cc:503] Master 127.15.102.254:38127 was elected leader, sending a full tablet report...
I20260620 18:44:43.046317 20892 raft_consensus.cc:374] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:43.046418 20892 raft_consensus.cc:729] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd2f8b22ab94b578855028d186b513d, State: Initialized, Role: FOLLOWER
I20260620 18:44:43.046694 20743 heartbeater.cc:503] Master 127.15.102.254:38127 was elected leader, sending a full tablet report...
I20260620 18:44:43.047079 20891 ts_tablet_manager.cc:1428] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260620 18:44:43.046940 20892 consensus_queue.cc:260] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.048872 20894 ts_tablet_manager.cc:1428] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260620 18:44:43.049760 20873 heartbeater.cc:503] Master 127.15.102.254:38127 was elected leader, sending a full tablet report...
I20260620 18:44:43.050285 20892 ts_tablet_manager.cc:1428] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260620 18:44:43.075145 20613 tablet.cc:2369] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:43.104704 20899 raft_consensus.cc:482] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:43.105021 20899 raft_consensus.cc:504] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.106663 20899 leader_election.cc:288] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bee0c40ddc4445da142f3ada67e43a1 (127.15.102.193:42775), b7b4ec2ce5aa4b93b459289ac2bc9b1a (127.15.102.194:33537)
I20260620 18:44:43.109331 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.107203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:51261 (local address 127.15.102.193:42775)
0620 18:44:43.107385 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:43.107394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:43.107414 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:44:43.107600 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:43.107605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:43.107663 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:43.107765 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.108736 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.108742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.108746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.108764 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.108801 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.108812 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.108933 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.109027 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.109082 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:44:43.109330 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.107117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.15.102.195:51261)
0620 18:44:43.107340 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.107360 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:43.107470 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:43.107793 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:43.107797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:43.107820 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:43.108516 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:44:43.108521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:43.108542 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:44:43.108588 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:43.108903 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:43.108907 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:43.108915 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:44:43.109039 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 18:44:43.109086 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:44:43.110354 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.107803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:39597 (local address 127.15.102.194:33537)
0620 18:44:43.107951 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:43.107956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:43.108449 (+ 493us) server_negotiation.cc:407] Connection header received
0620 18:44:43.108586 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:43.108590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:43.108637 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:43.108757 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.109752 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.109758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.109762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.109783 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.109829 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.109841 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.109967 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.110036 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.110092 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 18:44:43.110317 20567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6261cae391d485fb38908b6f1da419c" candidate_uuid: "9fd2f8b22ab94b578855028d186b513d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bee0c40ddc4445da142f3ada67e43a1" is_pre_election: true
I20260620 18:44:43.110544 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.107561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:33537 (local address 127.15.102.195:39597)
0620 18:44:43.108336 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.108352 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:43.108470 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:43.108777 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:43.108782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:43.108806 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:43.109499 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:44:43.109504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:43.109520 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:44:43.109567 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:43.109934 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:43.109938 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:43.109944 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:44:43.110060 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 18:44:43.110104 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":436,"threads_started":1}
I20260620 18:44:43.110836 20567 raft_consensus.cc:2454] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd2f8b22ab94b578855028d186b513d in term 0.
I20260620 18:44:43.111588 20762 leader_election.cc:302] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bee0c40ddc4445da142f3ada67e43a1, 9fd2f8b22ab94b578855028d186b513d; no voters:
I20260620 18:44:43.111507 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6261cae391d485fb38908b6f1da419c" candidate_uuid: "9fd2f8b22ab94b578855028d186b513d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" is_pre_election: true
I20260620 18:44:43.111974 20899 raft_consensus.cc:2781] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:43.112008 20698 raft_consensus.cc:2454] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd2f8b22ab94b578855028d186b513d in term 0.
I20260620 18:44:43.112130 20899 raft_consensus.cc:482] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:43.112229 20899 raft_consensus.cc:3037] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:43.114418 20899 raft_consensus.cc:504] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.114897 20899 leader_election.cc:288] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [CANDIDATE]: Term 1 election: Requested vote from peers 6bee0c40ddc4445da142f3ada67e43a1 (127.15.102.193:42775), b7b4ec2ce5aa4b93b459289ac2bc9b1a (127.15.102.194:33537)
I20260620 18:44:43.115451 20698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6261cae391d485fb38908b6f1da419c" candidate_uuid: "9fd2f8b22ab94b578855028d186b513d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a"
I20260620 18:44:43.115464 20567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6261cae391d485fb38908b6f1da419c" candidate_uuid: "9fd2f8b22ab94b578855028d186b513d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bee0c40ddc4445da142f3ada67e43a1"
I20260620 18:44:43.115660 20698 raft_consensus.cc:3037] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:43.115660 20567 raft_consensus.cc:3037] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:43.118870 20567 raft_consensus.cc:2454] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd2f8b22ab94b578855028d186b513d in term 1.
I20260620 18:44:43.118870 20698 raft_consensus.cc:2454] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd2f8b22ab94b578855028d186b513d in term 1.
I20260620 18:44:43.119321 20762 leader_election.cc:302] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fd2f8b22ab94b578855028d186b513d, b7b4ec2ce5aa4b93b459289ac2bc9b1a; no voters:
I20260620 18:44:43.119578 20899 raft_consensus.cc:2781] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:43.120012 20899 raft_consensus.cc:686] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 1 LEADER]: Becoming Leader. State: Replica: 9fd2f8b22ab94b578855028d186b513d, State: Running, Role: LEADER
I20260620 18:44:43.120468 20899 consensus_queue.cc:237] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } }
I20260620 18:44:43.124833 20424 catalog_manager.cc:5495] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fd2f8b22ab94b578855028d186b513d (127.15.102.195). New cstate: current_term: 1 leader_uuid: "9fd2f8b22ab94b578855028d186b513d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fd2f8b22ab94b578855028d186b513d" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34185 } health_report { overall_health: HEALTHY } } }
I20260620 18:44:43.142295 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.140519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.15.102.195:34185)
0620 18:44:43.140684 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:43.140688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:43.140777 (+ 89us) server_negotiation.cc:407] Connection header received
0620 18:44:43.140946 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:43.140949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:43.141026 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:43.141115 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.141880 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.141884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.141887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.141901 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.141930 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.141942 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.142038 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.142069 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.142109 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:44:43.150525 20698 raft_consensus.cc:1261] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 1 FOLLOWER]: Refusing update from remote peer 9fd2f8b22ab94b578855028d186b513d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:43.150524 20567 raft_consensus.cc:1261] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1 [term 1 FOLLOWER]: Refusing update from remote peer 9fd2f8b22ab94b578855028d186b513d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:43.151563 20901 consensus_queue.cc:1035] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bee0c40ddc4445da142f3ada67e43a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 42775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:43.152004 20899 consensus_queue.cc:1035] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4ec2ce5aa4b93b459289ac2bc9b1a" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:43.176808 20424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:43.177084 20424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:43.180470 20424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0daa83527ec34d29b99531bc4204fe62: Sending DeleteTablet for 3 replicas of tablet a6261cae391d485fb38908b6f1da419c
I20260620 18:44:43.181659 20808 tablet_service.cc:1514] Processing DeleteTablet for tablet a6261cae391d485fb38908b6f1da419c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:43 UTC) from {username='slave'} at 127.0.0.1:35024
I20260620 18:44:43.181692 20547 tablet_service.cc:1514] Processing DeleteTablet for tablet a6261cae391d485fb38908b6f1da419c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:43 UTC) from {username='slave'} at 127.0.0.1:57038
I20260620 18:44:43.181725 20678 tablet_service.cc:1514] Processing DeleteTablet for tablet a6261cae391d485fb38908b6f1da419c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:43 UTC) from {username='slave'} at 127.0.0.1:36660
I20260620 18:44:43.184430 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 20485
I20260620 18:44:43.200735 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.185597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.15.102.193:42775)
0620 18:44:43.185766 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:43.185772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:43.195944 (+ 10172us) server_negotiation.cc:407] Connection header received
0620 18:44:43.197111 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:43.197120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:43.197270 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:43.197360 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.199428 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.199437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.199444 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.199473 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.199525 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.199547 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.199689 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.200177 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.200258 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:44:43.202378 20607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:43.205327 20916 tablet_replica.cc:330] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: stopping tablet replica
I20260620 18:44:43.205829 20916 raft_consensus.cc:2229] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:43.206457 20915 tablet_replica.cc:330] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: stopping tablet replica
I20260620 18:44:43.206533 20916 raft_consensus.cc:2258] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:43.206900 20915 raft_consensus.cc:2229] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:43.208382 20915 raft_consensus.cc:2258] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:43.208755 20916 ts_tablet_manager.cc:1905] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:43.210315 20915 ts_tablet_manager.cc:1905] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:43.216307 20916 ts_tablet_manager.cc:1918] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:43.216578 20916 log.cc:1196] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/a6261cae391d485fb38908b6f1da419c
I20260620 18:44:43.217156 20916 ts_tablet_manager.cc:1939] T a6261cae391d485fb38908b6f1da419c P b7b4ec2ce5aa4b93b459289ac2bc9b1a: Deleting consensus metadata
I20260620 18:44:43.218550 20915 ts_tablet_manager.cc:1918] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:43.218828 20915 log.cc:1196] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/a6261cae391d485fb38908b6f1da419c
I20260620 18:44:43.219157 20411 catalog_manager.cc:4841] TS b7b4ec2ce5aa4b93b459289ac2bc9b1a (127.15.102.194:33537): tablet a6261cae391d485fb38908b6f1da419c (table test-table [id=90e187bbab494199bf1cd9ce92df2304]) successfully deleted
I20260620 18:44:43.219555 20915 ts_tablet_manager.cc:1939] T a6261cae391d485fb38908b6f1da419c P 9fd2f8b22ab94b578855028d186b513d: Deleting consensus metadata
I20260620 18:44:43.221354 20410 catalog_manager.cc:4841] TS 9fd2f8b22ab94b578855028d186b513d (127.15.102.195:34185): tablet a6261cae391d485fb38908b6f1da419c (table test-table [id=90e187bbab494199bf1cd9ce92df2304]) successfully deleted
I20260620 18:44:43.295806 20914 tablet_replica.cc:330] T a6261cae391d485fb38908b6f1da419c P 6bee0c40ddc4445da142f3ada67e43a1: stopping tablet replica
I20260620 18:44:43.296079 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 20485
I20260620 18:44:43.316627 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 20616
W20260620 18:44:43.316639 20411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:42775 (error 108)
I20260620 18:44:43.319123 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.318608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.0.0.1:57060)
0620 18:44:43.318812 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.318944 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 18:44:43.319420 20411 catalog_manager.cc:4584] TS 6bee0c40ddc4445da142f3ada67e43a1 (127.15.102.193:42775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6261cae391d485fb38908b6f1da419c: Network error: Client connection negotiation failed: client connection to 127.15.102.193:42775: connect: Connection refused (error 111)
I20260620 18:44:43.320322 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.317803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.15.102.194:33537)
0620 18:44:43.317995 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:43.318001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:43.318021 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:44:43.318087 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:43.318092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:43.318194 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:43.318306 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.319701 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.319707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.319712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.319738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.319768 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.319780 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.319914 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.320006 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.320064 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:44:43.321112 20738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:43.373385 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.372808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.0.0.1:57062)
0620 18:44:43.373083 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.373205 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 18:44:43.374590 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.374182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.0.0.1:57064)
0620 18:44:43.374349 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.374437 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:44:43.408202 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 20616
I20260620 18:44:43.425794 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 20746
I20260620 18:44:43.428908 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.426633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.15.102.195:34185)
0620 18:44:43.426833 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:43.426841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:43.426888 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:44:43.427053 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:43.427060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:43.427133 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:43.427294 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:43.428216 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:43.428222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:43.428227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:43.428244 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:43.428282 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:43.428297 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:43.428449 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:43.428510 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 18:44:43.428573 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:44:43.429587 20868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:43.435854 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.435456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.0.0.1:57068)
0620 18:44:43.435635 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.435726 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 18:44:43.436789 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.436445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.0.0.1:57076)
0620 18:44:43.436589 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.436672 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:44:43.517138 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.516686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.0.0.1:57080)
0620 18:44:43.516894 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.516990 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:44:43.518116 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:43.517757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:42775 (local address 127.0.0.1:57090)
0620 18:44:43.517948 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:43.518011 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:42775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:44:43.546833 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 20746
I20260620 18:44:43.568109 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 20395
I20260620 18:44:43.569170 20456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:43.673856 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 20395
2026-06-20T18:44:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2586 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 18:44:43.715197 15771 mini_oidc.cc:152] Starting JWKS server
I20260620 18:44:43.717861 15771 webserver.cc:460] Webserver started at https://127.0.0.1:37015/ using document root <none> and password file <none>
I20260620 18:44:43.718194 15771 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:44:43.718833 15771 webserver.cc:460] Webserver started at http://127.0.0.1:40407/ using document root <none> and password file <none>
2026-06-20T18:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:43Z Disabled control of system clock
I20260620 18:44:43.745016 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36881
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:36881
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37015/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/testchainca.cert with env {}
W20260620 18:44:44.061303 20928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:44.061578 20928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:44.061662 20928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:44.068624 20928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:44.068730 20928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:44.068768 20928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:44.068814 20928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:44.068864 20928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:44:44.068923 20928 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:44:44.068956 20928 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37015/jwks.json
I20260620 18:44:44.078753 20928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:36881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37015/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:44.080027 20928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:44.081297 20928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:44.093365 20936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:44.093410 20933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:44.093439 20934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:44.094626 20928 server_base.cc:1034] running on GCE node
I20260620 18:44:44.095517 20928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:44.096969 20928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:44.098202 20928 hybrid_clock.cc:648] HybridClock initialized: now 1781981084098151 us; error 40 us; skew 500 ppm
I20260620 18:44:44.098562 20928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:44:44.106822 20928 webserver.cc:460] Webserver started at http://127.15.102.254:34419/ using document root <none> and password file <none>
I20260620 18:44:44.107627 20928 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:44.107766 20928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:44.108074 20928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:44.110971 20928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "b50c5503b9684df8bdda8181a378e590"
format_stamp: "Formatted at 2026-06-20 18:44:44 on dist-test-slave-zfp8"
server_key: "c9e8ff9585fcf50141624d59483516ee"
server_key_iv: "38a2aa2c007534a879fb9aab61ca137a"
server_key_version: "encryptionkey@0"
I20260620 18:44:44.111871 20928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "b50c5503b9684df8bdda8181a378e590"
format_stamp: "Formatted at 2026-06-20 18:44:44 on dist-test-slave-zfp8"
server_key: "c9e8ff9585fcf50141624d59483516ee"
server_key_iv: "38a2aa2c007534a879fb9aab61ca137a"
server_key_version: "encryptionkey@0"
I20260620 18:44:44.116977 20928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:44:44.120589 20942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:44.121969 20928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:44.122156 20928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "b50c5503b9684df8bdda8181a378e590"
format_stamp: "Formatted at 2026-06-20 18:44:44 on dist-test-slave-zfp8"
server_key: "c9e8ff9585fcf50141624d59483516ee"
server_key_iv: "38a2aa2c007534a879fb9aab61ca137a"
server_key_version: "encryptionkey@0"
I20260620 18:44:44.122370 20928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:44.133204 20928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:44.133981 20928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:44.134279 20928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:44.152232 20928 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:36881
I20260620 18:44:44.152271 20993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:36881 every 8 connection(s)
I20260620 18:44:44.153548 20928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:44.157157 20994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:44.163359 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 20928
I20260620 18:44:44.163580 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:44.164207 15771 external_mini_cluster.cc:1428] Setting key e3c2d5bfafd6df2b6b486773621f3cc4
I20260620 18:44:44.167686 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590: Bootstrap starting.
I20260620 18:44:44.170926 20994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:44.171994 20994 log.cc:824] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:44.175096 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590: No bootstrap required, opened a new log
I20260620 18:44:44.178174 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:44.166066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.15.102.254:36881)
0620 18:44:44.167325 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:44.167337 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:44.167393 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:44:44.169203 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:44.169227 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:44.169666 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:44.169977 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:44.170561 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.171942 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.173650 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.174309 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.174469 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:44.175887 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:44.175894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:44.175909 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:44.176445 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:44.176518 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:44.176538 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:44.176629 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:44.176874 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:44:44.177215 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":374,"threads_started":1}
I20260620 18:44:44.181885 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:44:44.183703 20994 raft_consensus.cc:348] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50c5503b9684df8bdda8181a378e590" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36881 } }
I20260620 18:44:44.184118 20994 raft_consensus.cc:374] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:44.184221 20994 raft_consensus.cc:729] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50c5503b9684df8bdda8181a378e590, State: Initialized, Role: FOLLOWER
I20260620 18:44:44.184888 20994 consensus_queue.cc:260] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50c5503b9684df8bdda8181a378e590" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36881 } }
I20260620 18:44:44.185128 20994 raft_consensus.cc:388] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:44.185273 20994 raft_consensus.cc:482] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:44.185460 20994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:44.187641 20994 raft_consensus.cc:504] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50c5503b9684df8bdda8181a378e590" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36881 } }
I20260620 18:44:44.188102 20994 leader_election.cc:302] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b50c5503b9684df8bdda8181a378e590; no voters:
I20260620 18:44:44.188685 20994 leader_election.cc:288] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:44.188823 20999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:44.189550 20999 raft_consensus.cc:686] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [term 1 LEADER]: Becoming Leader. State: Replica: b50c5503b9684df8bdda8181a378e590, State: Running, Role: LEADER
I20260620 18:44:44.190001 20999 consensus_queue.cc:237] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50c5503b9684df8bdda8181a378e590" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36881 } }
I20260620 18:44:44.190738 20994 sys_catalog.cc:564] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:44.193248 21000 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b50c5503b9684df8bdda8181a378e590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50c5503b9684df8bdda8181a378e590" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36881 } } }
I20260620 18:44:44.192937 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b50c5503b9684df8bdda8181a378e590. Latest consensus state: current_term: 1 leader_uuid: "b50c5503b9684df8bdda8181a378e590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50c5503b9684df8bdda8181a378e590" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36881 } } }
I20260620 18:44:44.193856 21000 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:44.193883 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:44.195325 21007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:44.199815 21007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:44.207665 21007 catalog_manager.cc:1347] Generated new cluster ID: 93a3cf36deb144b18372736895a7123b
I20260620 18:44:44.207782 21007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:44.219208 21007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:44.221005 21007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:44.238103 21007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b50c5503b9684df8bdda8181a378e590: Generated new TSK 0
I20260620 18:44:44.239142 21007 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:44:44.278111 21024 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 18:44:44.278296 21024 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:44:44.279215 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:44.257891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.15.102.254:36881)
0620 18:44:44.258094 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:44.258100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:44.258577 (+ 477us) server_negotiation.cc:407] Connection header received
0620 18:44:44.258878 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:44.258884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:44.258982 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:44.259157 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:44:44.259719 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.260866 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.262262 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.262956 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.263088 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:44.263618 (+ 530us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:44:44.278554 (+ 14936us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:44:44.278923 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:44:44.279366 20997 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55802: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 18:44:44.280591 21023 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:44:44.257749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36881 (local address 127.0.0.1:55802)
0620 18:44:44.258458 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:44.258491 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:44.258637 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:44.259181 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:44.259184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:44.259194 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:44:44.259573 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:44.259583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.260998 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.261002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:44.262122 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:44.262133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.263124 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.263129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:44.263306 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:44.263365 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:44:44.278843 (+ 15478us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 18:44:44.279119 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.102.254:36881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":375,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.102.254:36881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 18:44:44.282393 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 20928
I20260620 18:44:44.283609 20989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:44.375922 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 20928
2026-06-20T18:44:44Z chronyd exiting
I20260620 18:44:44.426005 15771 test_util.cc:175] -----------------------------------------------
I20260620 18:44:44.426146 15771 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781981043007005-15771-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (715 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 18:44:44.429730 15771 mini_oidc.cc:152] Starting JWKS server
I20260620 18:44:44.431516 15771 webserver.cc:460] Webserver started at https://127.0.0.1:33651/ using document root <none> and password file <none>
I20260620 18:44:44.431699 15771 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:44:44.432278 15771 webserver.cc:460] Webserver started at http://127.0.0.1:40309/ using document root <none> and password file <none>
2026-06-20T18:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:44Z Disabled control of system clock
I20260620 18:44:44.460390 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:44359
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:44359
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33651/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/testchainca.cert with env {}
W20260620 18:44:44.775816 21031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:44.776135 21031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:44.776286 21031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:44.783583 21031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:44.783695 21031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:44.783784 21031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:44.783861 21031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:44.783923 21031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:44:44.784031 21031 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:44:44.784089 21031 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33651/jwks.json
I20260620 18:44:44.794044 21031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:44359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:44359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33651/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:44.795571 21031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:44.796954 21031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:44.808647 21036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:44.808594 21037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:44.808566 21039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:44.809487 21031 server_base.cc:1034] running on GCE node
I20260620 18:44:44.810346 21031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:44.811806 21031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:44.813058 21031 hybrid_clock.cc:648] HybridClock initialized: now 1781981084812999 us; error 57 us; skew 500 ppm
I20260620 18:44:44.813479 21031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:44:44.815696 21031 webserver.cc:460] Webserver started at http://127.15.102.254:40413/ using document root <none> and password file <none>
I20260620 18:44:44.816351 21031 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:44.816493 21031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:44.816795 21031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:44.821031 21031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "cafaa38e3d254feb946068e4ed03be2e"
format_stamp: "Formatted at 2026-06-20 18:44:44 on dist-test-slave-zfp8"
server_key: "9a27dc57e3a6b27360899e02759889db"
server_key_iv: "42b983e7fa0b9f17da3de6ee1f1d91c5"
server_key_version: "encryptionkey@0"
I20260620 18:44:44.821782 21031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "cafaa38e3d254feb946068e4ed03be2e"
format_stamp: "Formatted at 2026-06-20 18:44:44 on dist-test-slave-zfp8"
server_key: "9a27dc57e3a6b27360899e02759889db"
server_key_iv: "42b983e7fa0b9f17da3de6ee1f1d91c5"
server_key_version: "encryptionkey@0"
I20260620 18:44:44.826959 21031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:44:44.830382 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:44.831857 21031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:44:44.832038 21031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "cafaa38e3d254feb946068e4ed03be2e"
format_stamp: "Formatted at 2026-06-20 18:44:44 on dist-test-slave-zfp8"
server_key: "9a27dc57e3a6b27360899e02759889db"
server_key_iv: "42b983e7fa0b9f17da3de6ee1f1d91c5"
server_key_version: "encryptionkey@0"
I20260620 18:44:44.832267 21031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:44.855880 21031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:44.856693 21031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:44.856951 21031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:44.875026 21031 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:44359
I20260620 18:44:44.875051 21096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:44359 every 8 connection(s)
I20260620 18:44:44.876441 21031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:44.877898 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21031
I20260620 18:44:44.878082 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:44.878919 15771 external_mini_cluster.cc:1428] Setting key b00df67dc98c98594aa3b4285fb2a3f1
I20260620 18:44:44.881255 21097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:44.892155 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:44.880854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.15.102.254:44359)
0620 18:44:44.881879 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:44.881890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:44.881931 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:44:44.883560 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:44.883581 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:44.884054 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:44.884472 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:44.885055 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.886408 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.887764 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.888427 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.888631 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:44.889845 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:44.889853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:44.889870 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:44.890437 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:44.890509 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:44.890529 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:44.890671 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:44.890896 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 18:44:44.891182 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":356,"threads_started":1}
I20260620 18:44:44.892544 21097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e: Bootstrap starting.
I20260620 18:44:44.895900 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:44:44.895912 21097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:44.896875 21097 log.cc:824] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:44.899909 21097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e: No bootstrap required, opened a new log
I20260620 18:44:44.907562 21097 raft_consensus.cc:348] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafaa38e3d254feb946068e4ed03be2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 44359 } }
I20260620 18:44:44.907950 21097 raft_consensus.cc:374] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:44.908067 21097 raft_consensus.cc:729] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cafaa38e3d254feb946068e4ed03be2e, State: Initialized, Role: FOLLOWER
I20260620 18:44:44.908617 21097 consensus_queue.cc:260] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafaa38e3d254feb946068e4ed03be2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 44359 } }
I20260620 18:44:44.908840 21097 raft_consensus.cc:388] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:44.908986 21097 raft_consensus.cc:482] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:44.909157 21097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:44.911406 21097 raft_consensus.cc:504] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafaa38e3d254feb946068e4ed03be2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 44359 } }
I20260620 18:44:44.912012 21097 leader_election.cc:302] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cafaa38e3d254feb946068e4ed03be2e; no voters:
I20260620 18:44:44.912600 21097 leader_election.cc:288] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:44.912758 21102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:44.913463 21102 raft_consensus.cc:686] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [term 1 LEADER]: Becoming Leader. State: Replica: cafaa38e3d254feb946068e4ed03be2e, State: Running, Role: LEADER
I20260620 18:44:44.914050 21102 consensus_queue.cc:237] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafaa38e3d254feb946068e4ed03be2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 44359 } }
I20260620 18:44:44.914669 21097 sys_catalog.cc:564] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:44.917196 21103 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cafaa38e3d254feb946068e4ed03be2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafaa38e3d254feb946068e4ed03be2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 44359 } } }
I20260620 18:44:44.917685 21103 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:44.918203 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cafaa38e3d254feb946068e4ed03be2e. Latest consensus state: current_term: 1 leader_uuid: "cafaa38e3d254feb946068e4ed03be2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafaa38e3d254feb946068e4ed03be2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 44359 } } }
I20260620 18:44:44.918661 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:44.919878 21111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:44.924973 21111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:44.934204 21111 catalog_manager.cc:1347] Generated new cluster ID: 925d510eb1ab497e9a055a88a750a6dc
I20260620 18:44:44.934334 21111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:44.959813 21111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:44.961263 21111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:44.968271 21111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cafaa38e3d254feb946068e4ed03be2e: Generated new TSK 0
I20260620 18:44:44.969112 21111 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 18:44:45.004842 21127 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 18:44:45.004973 21127 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:44:45.006278 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:44.986056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.15.102.254:44359)
0620 18:44:44.986238 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:44.986244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:44.986562 (+ 318us) server_negotiation.cc:407] Connection header received
0620 18:44:44.986772 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:44.986777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:44.986863 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:44.987037 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:44:44.987537 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.988685 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.989968 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.990647 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.990776 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:44.991313 (+ 537us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:44:45.005553 (+ 14240us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:44:45.005937 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38540: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:44:45.006300 21126 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:44:44.985893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:44359 (local address 127.0.0.1:38540)
0620 18:44:44.986463 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:44.986478 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:44.986596 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:44.987047 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:44.987051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:44.987060 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:44:44.987368 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:44.987377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.988809 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.988812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:44.989833 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:44.989844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:44.990838 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:44.990843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:44.990983 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:44.991015 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:44:45.005806 (+ 14791us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 18:44:45.006005 (+ 199us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.102.254:44359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":324,"threads_started":1}
W20260620 18:44:45.006397 21100 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38540: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33651/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 18:44:45.007484 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21031
I20260620 18:44:45.008668 21092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:45.102849 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21031
2026-06-20T18:44:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (728 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 18:44:45.157999 15771 mini_oidc.cc:152] Starting JWKS server
I20260620 18:44:45.159951 15771 webserver.cc:460] Webserver started at https://127.0.0.1:43063/ using document root <none> and password file <none>
I20260620 18:44:45.160207 15771 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 18:44:45.160840 15771 webserver.cc:460] Webserver started at http://127.0.0.1:45051/ using document root <none> and password file <none>
2026-06-20T18:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:45Z Disabled control of system clock
I20260620 18:44:45.186713 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:38289
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:38289
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43063/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:44:45.504952 21134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:45.505239 21134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:45.505323 21134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:45.512645 21134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:45.512760 21134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:45.512796 21134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:45.512852 21134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:45.512895 21134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:44:45.512954 21134 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:44:45.513042 21134 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43063/jwks.json
I20260620 18:44:45.522221 21134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:38289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:38289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43063/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:45.523648 21134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:45.525393 21134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:45.536789 21139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:45.536913 21140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:45.536929 21142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:45.537707 21134 server_base.cc:1034] running on GCE node
I20260620 18:44:45.538527 21134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:45.540046 21134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:45.541337 21134 hybrid_clock.cc:648] HybridClock initialized: now 1781981085541288 us; error 54 us; skew 500 ppm
I20260620 18:44:45.541731 21134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:44:45.544217 21134 webserver.cc:460] Webserver started at http://127.15.102.254:34325/ using document root <none> and password file <none>
I20260620 18:44:45.544953 21134 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:45.545080 21134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:45.545423 21134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:45.548430 21134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "9f9b1f78f1ea4f41b970bff135d3fc03"
format_stamp: "Formatted at 2026-06-20 18:44:45 on dist-test-slave-zfp8"
server_key: "82ebb7f9d3a68a33b242f7fc0cab6f75"
server_key_iv: "0282b11efd0300943a5dea9bcf142a1f"
server_key_version: "encryptionkey@0"
I20260620 18:44:45.549232 21134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "9f9b1f78f1ea4f41b970bff135d3fc03"
format_stamp: "Formatted at 2026-06-20 18:44:45 on dist-test-slave-zfp8"
server_key: "82ebb7f9d3a68a33b242f7fc0cab6f75"
server_key_iv: "0282b11efd0300943a5dea9bcf142a1f"
server_key_version: "encryptionkey@0"
I20260620 18:44:45.554437 21134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:44:45.557951 21148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:45.559379 21134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:45.559557 21134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "9f9b1f78f1ea4f41b970bff135d3fc03"
format_stamp: "Formatted at 2026-06-20 18:44:45 on dist-test-slave-zfp8"
server_key: "82ebb7f9d3a68a33b242f7fc0cab6f75"
server_key_iv: "0282b11efd0300943a5dea9bcf142a1f"
server_key_version: "encryptionkey@0"
I20260620 18:44:45.559774 21134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:45.577593 21134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:45.578466 21134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:45.578774 21134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:45.597280 21134 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:38289
I20260620 18:44:45.597299 21199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:38289 every 8 connection(s)
I20260620 18:44:45.598670 21134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:45.602421 21200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:45.605103 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21134
I20260620 18:44:45.605324 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:45.606107 15771 external_mini_cluster.cc:1428] Setting key a8c19dd3f98ca0199868ddd62681455f
I20260620 18:44:45.614830 21200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03: Bootstrap starting.
I20260620 18:44:45.618458 21200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:45.619645 21200 log.cc:824] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:45.620116 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:45.608172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.15.102.254:38289)
0620 18:44:45.609235 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:45.609248 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:45.609290 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:44:45.610919 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:45.610941 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:45.611504 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:45.611858 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:45.612430 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:45.613707 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:45.615589 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:45.616244 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:45.616382 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:45.617630 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:45.617637 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:45.617650 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:45.618187 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:45.618280 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:45.618300 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:45.618396 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:45.618695 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 18:44:45.619040 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":424,"threads_started":1}
I20260620 18:44:45.622372 21200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03: No bootstrap required, opened a new log
I20260620 18:44:45.623865 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:44:45.630647 21200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9b1f78f1ea4f41b970bff135d3fc03" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38289 } }
I20260620 18:44:45.630963 21200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:45.631083 21200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9b1f78f1ea4f41b970bff135d3fc03, State: Initialized, Role: FOLLOWER
I20260620 18:44:45.631635 21200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9b1f78f1ea4f41b970bff135d3fc03" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38289 } }
I20260620 18:44:45.631825 21200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:45.631919 21200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:45.632031 21200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:45.634313 21200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9b1f78f1ea4f41b970bff135d3fc03" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38289 } }
I20260620 18:44:45.634750 21200 leader_election.cc:302] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f9b1f78f1ea4f41b970bff135d3fc03; no voters:
I20260620 18:44:45.635331 21200 leader_election.cc:288] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:45.635478 21205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:45.636166 21205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9b1f78f1ea4f41b970bff135d3fc03, State: Running, Role: LEADER
I20260620 18:44:45.636713 21205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9b1f78f1ea4f41b970bff135d3fc03" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38289 } }
I20260620 18:44:45.637452 21200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:45.639374 21206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f9b1f78f1ea4f41b970bff135d3fc03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9b1f78f1ea4f41b970bff135d3fc03" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38289 } } }
I20260620 18:44:45.639786 21206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:45.640676 21214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:45.640524 21207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f9b1f78f1ea4f41b970bff135d3fc03. Latest consensus state: current_term: 1 leader_uuid: "9f9b1f78f1ea4f41b970bff135d3fc03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9b1f78f1ea4f41b970bff135d3fc03" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 38289 } } }
I20260620 18:44:45.640826 21207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:45.645792 21214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:45.654507 21214 catalog_manager.cc:1347] Generated new cluster ID: ac9cffcae89046cc8917da8ff741bcdc
I20260620 18:44:45.654640 21214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:45.667025 21214 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:45.668340 21214 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:45.674296 21214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f9b1f78f1ea4f41b970bff135d3fc03: Generated new TSK 0
I20260620 18:44:45.675194 21214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:44:45.696849 21203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:45.688211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.15.102.254:38289)
0620 18:44:45.688418 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:45.688427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:45.688934 (+ 507us) server_negotiation.cc:407] Connection header received
0620 18:44:45.689173 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:45.689176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:45.689243 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:45.689374 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:44:45.689964 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:45.690689 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:45.692746 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:45.693209 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:45.693309 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:45.693896 (+ 587us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:44:45.696243 (+ 2347us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43063/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:44:45.696557 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37580: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43063/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:44:45.696980 21203 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37580: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43063/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 18:44:45.699026 21229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:44:45.688206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:38289 (local address 127.0.0.1:37580)
0620 18:44:45.688828 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:45.688847 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:45.688999 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:45.689420 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:45.689425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:45.689439 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:44:45.689803 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:45.689816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:45.690830 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:45.690835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:45.692583 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:45.692598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:45.693416 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:45.693423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:45.693605 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:45.693652 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:44:45.696461 (+ 2809us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43063/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 18:44:45.698561 (+ 2100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.102.254:38289: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43063/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":1799,"thread_start_us":347,"threads_started":1}
I20260620 18:44:45.700112 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21134
I20260620 18:44:45.701316 21195 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 18:44:45.713019 21230 webserver.cc:533] Webserver: sslize failed:
E20260620 18:44:45.713166 21230 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 18:44:45.792699 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21134
2026-06-20T18:44:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (689 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T18:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:45Z Disabled control of system clock
I20260620 18:44:45.874600 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37571
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:34553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:37571
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/testchainca.cert with env {}
W20260620 18:44:46.193011 21235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:46.193321 21235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:46.193480 21235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:46.200693 21235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:46.200809 21235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:46.200942 21235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:46.201045 21235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:46.201113 21235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 18:44:46.201197 21235 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 18:44:46.201272 21235 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 18:44:46.211469 21235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:37571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:46.212908 21235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:46.214236 21235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:46.226249 21240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:46.226217 21241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:46.226176 21243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:46.227166 21235 server_base.cc:1034] running on GCE node
I20260620 18:44:46.227980 21235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:46.229382 21235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:46.230583 21235 hybrid_clock.cc:648] HybridClock initialized: now 1781981086230530 us; error 53 us; skew 500 ppm
I20260620 18:44:46.230944 21235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:44:46.233404 21235 webserver.cc:460] Webserver started at http://127.15.102.254:40599/ using document root <none> and password file <none>
I20260620 18:44:46.234107 21235 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:46.234223 21235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:46.234544 21235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:46.237425 21235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "29f724fc382940b29c515c9ccab316b8"
format_stamp: "Formatted at 2026-06-20 18:44:46 on dist-test-slave-zfp8"
server_key: "25c0505da78114a555ee2a25e56f8157"
server_key_iv: "209137dc80cab56818f9ab15ba5fd940"
server_key_version: "encryptionkey@0"
I20260620 18:44:46.238219 21235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "29f724fc382940b29c515c9ccab316b8"
format_stamp: "Formatted at 2026-06-20 18:44:46 on dist-test-slave-zfp8"
server_key: "25c0505da78114a555ee2a25e56f8157"
server_key_iv: "209137dc80cab56818f9ab15ba5fd940"
server_key_version: "encryptionkey@0"
I20260620 18:44:46.243482 21235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260620 18:44:46.246793 21249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:46.248299 21235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:46.248528 21235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "29f724fc382940b29c515c9ccab316b8"
format_stamp: "Formatted at 2026-06-20 18:44:46 on dist-test-slave-zfp8"
server_key: "25c0505da78114a555ee2a25e56f8157"
server_key_iv: "209137dc80cab56818f9ab15ba5fd940"
server_key_version: "encryptionkey@0"
I20260620 18:44:46.248752 21235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:46.270002 21235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:46.270864 21235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:46.271148 21235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:46.289186 21235 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:37571
I20260620 18:44:46.289193 21300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:37571 every 8 connection(s)
I20260620 18:44:46.290563 21235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:46.293772 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21235
I20260620 18:44:46.293979 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:46.294450 21301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:46.294669 15771 external_mini_cluster.cc:1428] Setting key 0fea7a778dab3e8f7fc4000fcf45ab7d
I20260620 18:44:46.309558 21301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8: Bootstrap starting.
I20260620 18:44:46.309713 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:46.296558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.15.102.254:37571)
0620 18:44:46.297651 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:46.297665 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:46.297718 (+ 53us) server_negotiation.cc:407] Connection header received
0620 18:44:46.299516 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:46.299543 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:46.300162 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:46.300617 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:46.301160 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:46.303167 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:46.304519 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:46.305559 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:46.305747 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:46.306881 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:46.306891 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:46.306909 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:44:46.307659 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:46.307750 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:44:46.307774 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:46.307879 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:46.308099 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 18:44:46.308415 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":315,"threads_started":1}
I20260620 18:44:46.313462 21301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:46.313994 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:44:46.314779 21301 log.cc:824] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:46.317842 21301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8: No bootstrap required, opened a new log
I20260620 18:44:46.325117 21301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f724fc382940b29c515c9ccab316b8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37571 } }
I20260620 18:44:46.325493 21301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:46.325625 21301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f724fc382940b29c515c9ccab316b8, State: Initialized, Role: FOLLOWER
I20260620 18:44:46.326285 21301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f724fc382940b29c515c9ccab316b8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37571 } }
I20260620 18:44:46.326562 21301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:46.326717 21301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:46.326895 21301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:46.329053 21301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f724fc382940b29c515c9ccab316b8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37571 } }
I20260620 18:44:46.329525 21301 leader_election.cc:302] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29f724fc382940b29c515c9ccab316b8; no voters:
I20260620 18:44:46.330054 21301 leader_election.cc:288] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:46.330217 21306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:46.330897 21306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [term 1 LEADER]: Becoming Leader. State: Replica: 29f724fc382940b29c515c9ccab316b8, State: Running, Role: LEADER
I20260620 18:44:46.331437 21306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f724fc382940b29c515c9ccab316b8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37571 } }
I20260620 18:44:46.332224 21301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:46.334077 21308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29f724fc382940b29c515c9ccab316b8. Latest consensus state: current_term: 1 leader_uuid: "29f724fc382940b29c515c9ccab316b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f724fc382940b29c515c9ccab316b8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37571 } } }
I20260620 18:44:46.334437 21308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:46.334973 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29f724fc382940b29c515c9ccab316b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29f724fc382940b29c515c9ccab316b8" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37571 } } }
I20260620 18:44:46.335314 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:46.335493 21314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:46.341356 21314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:46.350039 21314 catalog_manager.cc:1347] Generated new cluster ID: d214ca7fd95442ecb3442961f3e7a8bc
I20260620 18:44:46.350172 21314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:46.360316 21314 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:46.361550 21314 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:46.372342 21314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29f724fc382940b29c515c9ccab316b8: Generated new TSK 0
I20260620 18:44:46.373253 21314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:44:46.516132 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:46.384945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.15.102.254:37571)
0620 18:44:46.385142 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:46.385148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:46.385572 (+ 424us) server_negotiation.cc:407] Connection header received
0620 18:44:46.385788 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:46.385793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:46.385889 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:46.386055 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0620 18:44:46.386451 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:46.387592 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:46.388924 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:46.389662 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:46.389790 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:46.390253 (+ 463us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 18:44:46.515377 (+125124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:44:46.515804 (+ 427us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60548: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 18:44:46.516286 21304 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60548: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 18:44:46.516352 21330 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:44:46.384775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37571 (local address 127.0.0.1:60548)
0620 18:44:46.385462 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:46.385478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:46.385600 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:46.386067 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:46.386070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:46.386080 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 18:44:46.386306 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:46.386313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:46.387734 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:46.387737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:46.388786 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:46.388797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:46.389824 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:46.389828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:46.389950 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:46.389982 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 18:44:46.515701 (+125719us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 18:44:46.515971 (+ 270us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.102.254:37571: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":394,"threads_started":1}
I20260620 18:44:46.517768 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21235
I20260620 18:44:46.519124 21296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:46.621254 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21235
2026-06-20T18:44:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (808 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (401 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (405 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:47 dist-test-slave-zfp8 krb5kdc[21345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:47 dist-test-slave-zfp8 krb5kdc[21345](info): set up 2 sockets
Jun 20 18:44:47 dist-test-slave-zfp8 krb5kdc[21345](info): commencing operation
krb5kdc: starting...
W20260620 18:44:48.507871 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:48 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:44:48.707418 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:45639
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:45639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:49.028021 21358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:49.028303 21358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:49.028477 21358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:49.035648 21358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:49.035750 21358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:49.035845 21358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:49.035945 21358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:49.036023 21358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:49.045740 21358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:45639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:45639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:49.047108 21358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:49.048447 21358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:49.060272 21363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:49.060235 21366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:49.060312 21364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:49.061508 21358 server_base.cc:1034] running on GCE node
I20260620 18:44:49.062321 21358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:49.063758 21358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:49.064970 21358 hybrid_clock.cc:648] HybridClock initialized: now 1781981089064928 us; error 34 us; skew 500 ppm
I20260620 18:44:49.065304 21358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:49 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981089, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:49.068331 21358 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:44:49.069875 21358 webserver.cc:460] Webserver started at http://127.15.102.254:39081/ using document root <none> and password file <none>
I20260620 18:44:49.070554 21358 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:49.070700 21358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:49.071025 21358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:49.073947 21358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "df30939eea0c4f288a4a8666add2f2a3"
format_stamp: "Formatted at 2026-06-20 18:44:49 on dist-test-slave-zfp8"
server_key: "c79209e851e9e0e177d11cbc8bd97321"
server_key_iv: "06a7a58ad8846555d415440497b69492"
server_key_version: "encryptionkey@0"
I20260620 18:44:49.074719 21358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "df30939eea0c4f288a4a8666add2f2a3"
format_stamp: "Formatted at 2026-06-20 18:44:49 on dist-test-slave-zfp8"
server_key: "c79209e851e9e0e177d11cbc8bd97321"
server_key_iv: "06a7a58ad8846555d415440497b69492"
server_key_version: "encryptionkey@0"
I20260620 18:44:49.079710 21358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:44:49.082847 21373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:49.084255 21358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:44:49.084451 21358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "df30939eea0c4f288a4a8666add2f2a3"
format_stamp: "Formatted at 2026-06-20 18:44:49 on dist-test-slave-zfp8"
server_key: "c79209e851e9e0e177d11cbc8bd97321"
server_key_iv: "06a7a58ad8846555d415440497b69492"
server_key_version: "encryptionkey@0"
I20260620 18:44:49.084645 21358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:49.117440 21358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:49.121650 21358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:49.121956 21358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:49.140309 21358 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:45639
I20260620 18:44:49.140342 21424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:45639 every 8 connection(s)
I20260620 18:44:49.141645 21358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:49.145159 21425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:49.146757 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21358
I20260620 18:44:49.146967 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:49.147655 15771 external_mini_cluster.cc:1428] Setting key edb823c27bc3cacb5dfb3696a1f3590b
I20260620 18:44:49.156565 21425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3: Bootstrap starting.
I20260620 18:44:49.160239 21425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:49.161428 21425 log.cc:824] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:44:49 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:49.164820 21425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3: No bootstrap required, opened a new log
I20260620 18:44:49.171025 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:49.149540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.15.102.254:45639)
0620 18:44:49.150292 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:49.150303 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:49.150345 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:44:49.152016 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:49.152037 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:49.152392 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:49.152712 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:49.154359 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.155656 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.157319 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.157952 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.158078 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:49.162931 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:49.162941 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:49.162959 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:49.163005 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:49.166734 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.167371 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.167381 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.167390 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.167608 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.167962 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.167968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.167971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.168659 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:49.168932 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:49.169288 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 18:44:49.169703 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":350,"threads_started":1}
I20260620 18:44:49.174763 21425 raft_consensus.cc:348] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df30939eea0c4f288a4a8666add2f2a3" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45639 } }
I20260620 18:44:49.175074 21425 raft_consensus.cc:374] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:49.175196 21425 raft_consensus.cc:729] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df30939eea0c4f288a4a8666add2f2a3, State: Initialized, Role: FOLLOWER
I20260620 18:44:49.175828 21425 consensus_queue.cc:260] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df30939eea0c4f288a4a8666add2f2a3" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45639 } }
I20260620 18:44:49.176064 21425 raft_consensus.cc:388] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:49.176206 21425 raft_consensus.cc:482] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:49.176375 21425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:49.178443 21425 raft_consensus.cc:504] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df30939eea0c4f288a4a8666add2f2a3" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45639 } }
I20260620 18:44:49.178912 21425 leader_election.cc:302] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df30939eea0c4f288a4a8666add2f2a3; no voters:
I20260620 18:44:49.179623 21425 leader_election.cc:288] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:49.179765 21430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:49.180581 21430 raft_consensus.cc:686] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [term 1 LEADER]: Becoming Leader. State: Replica: df30939eea0c4f288a4a8666add2f2a3, State: Running, Role: LEADER
I20260620 18:44:49.181071 21430 consensus_queue.cc:237] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df30939eea0c4f288a4a8666add2f2a3" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45639 } }
I20260620 18:44:49.181595 21425 sys_catalog.cc:564] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:49.184221 21431 sys_catalog.cc:455] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df30939eea0c4f288a4a8666add2f2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df30939eea0c4f288a4a8666add2f2a3" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45639 } } }
I20260620 18:44:49.184762 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df30939eea0c4f288a4a8666add2f2a3. Latest consensus state: current_term: 1 leader_uuid: "df30939eea0c4f288a4a8666add2f2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df30939eea0c4f288a4a8666add2f2a3" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45639 } } }
I20260620 18:44:49.185055 21431 sys_catalog.cc:458] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:49.185114 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:49.186808 21441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:49.191455 21441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:49.199301 21441 catalog_manager.cc:1347] Generated new cluster ID: 40031b3314e14c02972acc4137751b5c
I20260620 18:44:49.199412 21441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:49.229015 21441 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:49.230345 21441 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:49.244545 21441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df30939eea0c4f288a4a8666add2f2a3: Generated new TSK 0
I20260620 18:44:49.245352 21441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:44:49.294263 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45639
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:49.634472 21451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:49.634742 21451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:49.634832 21451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:49.642103 21451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:49.642210 21451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:49.642360 21451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:49.652396 21451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:49.653678 21451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:49.654940 21451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:49.668136 21456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:49.668187 21459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:49.668222 21457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:49.669443 21451 server_base.cc:1034] running on GCE node
I20260620 18:44:49.670279 21451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:49.671424 21451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:49.672677 21451 hybrid_clock.cc:648] HybridClock initialized: now 1781981089672627 us; error 40 us; skew 500 ppm
I20260620 18:44:49.673046 21451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:49 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981089, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:49.676275 21451 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:44:49.677776 21451 webserver.cc:460] Webserver started at http://127.15.102.193:33945/ using document root <none> and password file <none>
I20260620 18:44:49.678488 21451 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:49.678639 21451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:49.678952 21451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:49.682065 21451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "78b502d0bead45019492c11b30b5d585"
format_stamp: "Formatted at 2026-06-20 18:44:49 on dist-test-slave-zfp8"
server_key: "0ab849f85f7a7ed34146d795d4d230b3"
server_key_iv: "78af2f40f561faf90e03163e3610618b"
server_key_version: "encryptionkey@0"
I20260620 18:44:49.682964 21451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "78b502d0bead45019492c11b30b5d585"
format_stamp: "Formatted at 2026-06-20 18:44:49 on dist-test-slave-zfp8"
server_key: "0ab849f85f7a7ed34146d795d4d230b3"
server_key_iv: "78af2f40f561faf90e03163e3610618b"
server_key_version: "encryptionkey@0"
I20260620 18:44:49.688349 21451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:44:49.691810 21466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:49.693298 21451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:49.693431 21451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "78b502d0bead45019492c11b30b5d585"
format_stamp: "Formatted at 2026-06-20 18:44:49 on dist-test-slave-zfp8"
server_key: "0ab849f85f7a7ed34146d795d4d230b3"
server_key_iv: "78af2f40f561faf90e03163e3610618b"
server_key_version: "encryptionkey@0"
I20260620 18:44:49.693593 21451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:49.707490 21451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:49.711401 21451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:49.711642 21451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:49.712539 21451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:49.713845 21451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:49.713917 21451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:49.713997 21451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:49.714044 21451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:49.746134 21451 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:43357
I20260620 18:44:49.746176 21578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:43357 every 8 connection(s)
I20260620 18:44:49.747546 21451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:49.754243 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21451
I20260620 18:44:49.754442 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:49.755168 15771 external_mini_cluster.cc:1428] Setting key 209263d2755054f96b6cfdbffef81a99
Jun 20 18:44:49 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981089, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:49.767774 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:49.750046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:49473 (local address 127.15.102.254:45639)
0620 18:44:49.750527 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:49.750534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:49.751548 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 18:44:49.753247 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:49.753251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:49.753314 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:49.753408 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:49.754914 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.755965 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.757468 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.758159 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.758303 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:49.761674 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:49.761683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:49.761691 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:49.761735 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:49.764572 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.765506 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.765509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.765512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.765615 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:49.766082 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:49.766085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:49.766087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:49.766321 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:49.766435 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:49.767349 (+ 914us) server_negotiation.cc:299] Negotiation successful
0620 18:44:49.767534 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
I20260620 18:44:49.769322 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:49.750627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45639 (local address 127.15.102.193:49473)
0620 18:44:49.751396 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:49.751439 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:49.753009 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:49.753606 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:49.753615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:49.754103 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:49.754748 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:49.754766 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.756121 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.756125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:49.757322 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:49.757333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:49.758363 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:49.758368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:49.758591 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:49.759344 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:49.759366 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:49.761450 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:49.764721 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:49.764733 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:49.764754 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:49.765364 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:49.765722 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:49.765729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:49.765735 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:49.765978 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:49.766457 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:49.766467 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:49.766917 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 18:44:49.767216 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 18:44:49.767826 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":278,"threads_started":1}
I20260620 18:44:49.771088 21579 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45639
I20260620 18:44:49.771448 21579 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:49.772279 21579 heartbeater.cc:511] Master 127.15.102.254:45639 requested a full tablet report, sending...
I20260620 18:44:49.774680 21390 ts_manager.cc:194] Registered new tserver with Master: 78b502d0bead45019492c11b30b5d585 (127.15.102.193:43357)
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:44:49.776724 21390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:49473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:44:49.800612 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45639
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:50.119176 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:50.119479 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:50.119571 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:50.126602 21585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:50.126722 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:50.126834 21585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:50.136591 21585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:50.137881 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:50.139294 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:50.153031 21590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:50.153223 21591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:50.154233 21593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:50.155321 21585 server_base.cc:1034] running on GCE node
I20260620 18:44:50.155912 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:50.156878 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:50.158107 21585 hybrid_clock.cc:648] HybridClock initialized: now 1781981090158051 us; error 49 us; skew 500 ppm
I20260620 18:44:50.158413 21585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.161520 21585 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:44:50.163013 21585 webserver.cc:460] Webserver started at http://127.15.102.194:35747/ using document root <none> and password file <none>
I20260620 18:44:50.163739 21585 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:50.163836 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:50.164085 21585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:50.166817 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "ab13dda77a26490eab704a1a57f2db38"
format_stamp: "Formatted at 2026-06-20 18:44:50 on dist-test-slave-zfp8"
server_key: "11bb4974f21e8c07e93325160ef3d428"
server_key_iv: "9a3ca03421cae75a5bdd3063994a14a5"
server_key_version: "encryptionkey@0"
I20260620 18:44:50.167663 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "ab13dda77a26490eab704a1a57f2db38"
format_stamp: "Formatted at 2026-06-20 18:44:50 on dist-test-slave-zfp8"
server_key: "11bb4974f21e8c07e93325160ef3d428"
server_key_iv: "9a3ca03421cae75a5bdd3063994a14a5"
server_key_version: "encryptionkey@0"
I20260620 18:44:50.172667 21585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:44:50.176082 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.177419 21585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:50.177625 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "ab13dda77a26490eab704a1a57f2db38"
format_stamp: "Formatted at 2026-06-20 18:44:50 on dist-test-slave-zfp8"
server_key: "11bb4974f21e8c07e93325160ef3d428"
server_key_iv: "9a3ca03421cae75a5bdd3063994a14a5"
server_key_version: "encryptionkey@0"
I20260620 18:44:50.177889 21585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:50.201022 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:50.205320 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:50.205601 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:50.206532 21585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:50.207860 21585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:50.207937 21585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.208022 21585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:50.208065 21585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.239392 21585 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:34995
I20260620 18:44:50.239468 21712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:34995 every 8 connection(s)
I20260620 18:44:50.240701 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:50.249251 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21585
I20260620 18:44:50.249442 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:50.249992 15771 external_mini_cluster.cc:1428] Setting key 3b91635ed834a62dc3190f3c24d9fe02
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:50.262629 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.243337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:46143 (local address 127.15.102.254:45639)
0620 18:44:50.243507 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.243512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.244725 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 18:44:50.246593 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.246598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.246666 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.246758 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.248324 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.249045 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.251147 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.251806 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.251963 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.256507 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:50.256517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:50.256526 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:50.256576 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:50.259352 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.260079 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.260084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.260088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.260222 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.260651 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.260663 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.260665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.260987 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:50.261142 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:50.262139 (+ 997us) server_negotiation.cc:299] Negotiation successful
0620 18:44:50.262378 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260620 18:44:50.264185 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.243772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45639 (local address 127.15.102.194:46143)
0620 18:44:50.244485 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:50.244528 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:50.246323 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:50.246971 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:50.246980 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:50.247588 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:50.248160 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:50.248181 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.249213 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.249221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:50.251008 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:50.251020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.252031 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.252036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:50.252351 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.253151 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:50.253186 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:50.256198 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:50.259498 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:50.259507 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:50.259531 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:50.259946 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:50.260338 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:50.260344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:50.260347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:50.260530 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:50.261157 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:50.261168 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:50.261673 (+ 505us) client_negotiation.cc:769] Sending connection context
0620 18:44:50.261990 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 18:44:50.262597 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":285,"threads_started":1}
I20260620 18:44:50.265707 21713 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45639
I20260620 18:44:50.266023 21713 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:50.266750 21713 heartbeater.cc:511] Master 127.15.102.254:45639 requested a full tablet report, sending...
I20260620 18:44:50.268386 21390 ts_manager.cc:194] Registered new tserver with Master: ab13dda77a26490eab704a1a57f2db38 (127.15.102.194:34995)
I20260620 18:44:50.269202 21390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:46143
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:44:50.298679 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45639
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:50.635444 21718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:50.635718 21718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:50.635810 21718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:50.642987 21718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:50.643095 21718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:50.643206 21718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:50.652818 21718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:50.654032 21718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:50.655367 21718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:50.668381 21724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:50.668421 21723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:50.668455 21726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:50.669234 21718 server_base.cc:1034] running on GCE node
I20260620 18:44:50.669847 21718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:50.670855 21718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:50.672049 21718 hybrid_clock.cc:648] HybridClock initialized: now 1781981090671983 us; error 59 us; skew 500 ppm
I20260620 18:44:50.672356 21718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.675480 21718 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:44:50.676900 21718 webserver.cc:460] Webserver started at http://127.15.102.195:42403/ using document root <none> and password file <none>
I20260620 18:44:50.677582 21718 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:50.677726 21718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:50.678028 21718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:50.680964 21718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "dd2cf327550a43c280942e0930cdc89f"
format_stamp: "Formatted at 2026-06-20 18:44:50 on dist-test-slave-zfp8"
server_key: "8ae28f4f8550f83206e46180b65041ae"
server_key_iv: "7bc4f75f4f8631d9f2861698bdcffe44"
server_key_version: "encryptionkey@0"
I20260620 18:44:50.681834 21718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "dd2cf327550a43c280942e0930cdc89f"
format_stamp: "Formatted at 2026-06-20 18:44:50 on dist-test-slave-zfp8"
server_key: "8ae28f4f8550f83206e46180b65041ae"
server_key_iv: "7bc4f75f4f8631d9f2861698bdcffe44"
server_key_version: "encryptionkey@0"
I20260620 18:44:50.687151 21718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:44:50.690734 21733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.692152 21718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:44:50.692314 21718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "dd2cf327550a43c280942e0930cdc89f"
format_stamp: "Formatted at 2026-06-20 18:44:50 on dist-test-slave-zfp8"
server_key: "8ae28f4f8550f83206e46180b65041ae"
server_key_iv: "7bc4f75f4f8631d9f2861698bdcffe44"
server_key_version: "encryptionkey@0"
I20260620 18:44:50.692520 21718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:50.716032 21718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:50.720249 21718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:50.720485 21718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:50.721364 21718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:50.722596 21718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:50.722666 21718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.722747 21718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:50.722790 21718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:50.754873 21718 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:46515
I20260620 18:44:50.754923 21845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:46515 every 8 connection(s)
I20260620 18:44:50.756209 21718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:50.758858 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21718
I20260620 18:44:50.759047 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:50.759615 15771 external_mini_cluster.cc:1428] Setting key a0c8a565af7ad2182cce4baa9c7a6b84
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:44:50.776278 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.758916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:50501 (local address 127.15.102.254:45639)
0620 18:44:50.759181 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.759187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.760694 (+ 1507us) server_negotiation.cc:407] Connection header received
0620 18:44:50.762801 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.762805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.762892 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.763005 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.764562 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.765475 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.767005 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.767483 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.767612 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.770850 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:50.770855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:50.770858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:50.770889 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:50.773248 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.773950 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.773962 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.773965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.774079 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.774486 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.774491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.774495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.774771 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:50.774923 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:50.775712 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 18:44:50.775923 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:44:50.777266 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.759608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45639 (local address 127.15.102.195:50501)
0620 18:44:50.760537 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:50.760582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:50.762540 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:50.763290 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:50.763303 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:50.763831 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:50.764397 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:50.764414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.765622 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.765628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:50.766856 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:50.766868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.767708 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.767717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:50.767958 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.768586 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:50.768608 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:50.770645 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:50.773386 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:50.773395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:50.773411 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:50.773819 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:50.774186 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:50.774192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:50.774196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:50.774372 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:50.774904 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:50.774912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:50.775282 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 18:44:50.775592 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 18:44:50.776187 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":348,"threads_started":1}
I20260620 18:44:50.778556 21846 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45639
I20260620 18:44:50.778857 21846 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:50.779618 21846 heartbeater.cc:511] Master 127.15.102.254:45639 requested a full tablet report, sending...
I20260620 18:44:50.779877 21579 heartbeater.cc:503] Master 127.15.102.254:45639 was elected leader, sending a full tablet report...
I20260620 18:44:50.781213 21389 ts_manager.cc:194] Registered new tserver with Master: dd2cf327550a43c280942e0930cdc89f (127.15.102.195:46515)
I20260620 18:44:50.782086 21389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:50501
I20260620 18:44:50.787132 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/test-admin.keytab.
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.813156 15771 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 18:44:50.818285 21853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:44:50.815849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.15.102.193:43357)
0620 18:44:50.816502 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.816510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.816540 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:44:50.816606 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.816614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.816844 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.816988 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.817831 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57152: BlockingRecv error: recv got EOF from 127.0.0.1:57152 (error 108)
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":321,"threads_started":1}
W20260620 18:44:50.818289 15771 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.820118 15771 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.821403 15771 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:50.836277 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.823007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.15.102.193:43357)
0620 18:44:50.823200 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.823207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.823414 (+ 207us) server_negotiation.cc:407] Connection header received
0620 18:44:50.823584 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.823588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.823643 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.823721 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.824711 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.825681 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.826898 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.827525 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.827634 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.831018 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:50.831029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:50.831044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:50.831077 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:50.833834 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.834438 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.834445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.834447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.834566 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.834863 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.834866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.834868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.835186 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:50.835481 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:50.835748 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:44:50.836014 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 18:44:50.842037 21853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:44:50.840210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.15.102.193:43357)
0620 18:44:50.840431 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.840437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.840683 (+ 246us) server_negotiation.cc:407] Connection header received
0620 18:44:50.840887 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.840891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.840964 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.841063 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.841797 (+ 734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57174: BlockingRecv error: recv got EOF from 127.0.0.1:57174 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 18:44:50.842255 15771 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.843768 15771 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.844981 15771 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:50.858080 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.846250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.15.102.193:43357)
0620 18:44:50.846432 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.846437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.846613 (+ 176us) server_negotiation.cc:407] Connection header received
0620 18:44:50.846723 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.846726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.846776 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.846849 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.847816 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.848498 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.849623 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.850015 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.850110 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.853358 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:50.853363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:50.853366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:50.853408 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:50.856040 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.856563 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.856567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.856569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.856681 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.856972 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.856975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.856977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.857228 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:50.857411 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:50.857590 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 18:44:50.857732 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:44:50.861991 21853 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:44:50.860475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.15.102.193:43357)
0620 18:44:50.860662 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.860671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.860930 (+ 259us) server_negotiation.cc:407] Connection header received
0620 18:44:50.861034 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.861037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.861109 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.861183 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.861783 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57192: BlockingRecv error: recv got EOF from 127.0.0.1:57192 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 18:44:50.862149 15771 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781981043007005-15771-0/krb5kdc/krb5cc' of type FILE
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.863725 15771 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:50.864997 15771 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:44:50.878131 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.866218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.15.102.193:43357)
0620 18:44:50.866419 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.866424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.866599 (+ 175us) server_negotiation.cc:407] Connection header received
0620 18:44:50.866703 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.866706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.866759 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.866833 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.867788 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.868471 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.869607 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.870000 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.870090 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.873474 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:50.873478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:50.873482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:50.873512 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:50.875905 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.876441 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.876444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.876447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.876604 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.876903 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.876906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.876908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.877205 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:50.877415 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:50.877626 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 18:44:50.877810 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:44:50.879580 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21451
I20260620 18:44:50.891597 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.880347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.15.102.193:43357)
0620 18:44:50.880538 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.880544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.880752 (+ 208us) server_negotiation.cc:407] Connection header received
0620 18:44:50.880923 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.880928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.880988 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.881087 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.882067 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.882803 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.884024 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.884545 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.884708 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:50.886706 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:50.886711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:50.886714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:50.886741 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:50.889125 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.889728 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.889735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.889739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.889868 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:50.890225 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:50.890231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:50.890235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:50.890483 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:50.890737 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:50.891045 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 18:44:50.891268 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:44:50.892190 21574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:50.975150 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21451
I20260620 18:44:50.991750 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21585
Jun 20 18:44:50 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:44:51.006464 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:50.992644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.15.102.194:34995)
0620 18:44:50.993227 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:50.993233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:50.993258 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:44:50.993317 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:50.993323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:50.993535 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:50.993669 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:50.994538 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.995699 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.996976 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:50.997511 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:50.997607 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:51.001353 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:51.001364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:51.001378 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:51.001416 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:51.004095 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:51.004730 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:51.004738 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:51.004740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:51.004853 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:51.005180 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:51.005183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:51.005185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:51.005498 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:51.005719 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:51.005985 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 18:44:51.006181 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":309,"threads_started":1}
I20260620 18:44:51.007894 21708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:51.089972 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21585
I20260620 18:44:51.105751 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21718
Jun 20 18:44:51 dist-test-slave-zfp8 krb5kdc[21345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:44:51.121565 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:51.106654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.15.102.195:46515)
0620 18:44:51.107349 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:51.107356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:51.107386 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:44:51.107450 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:51.107457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:51.107674 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:51.107839 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:51.108849 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:51.109886 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:51.111057 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:51.111744 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:51.111841 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:51.115740 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:51.115751 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:51.115767 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:51.115802 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:51.118856 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:51.119868 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:51.119875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:51.119877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:51.120004 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:51.120305 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:51.120308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:51.120310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:51.120630 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:51.120853 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:51.121084 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:44:51.121269 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":328,"threads_started":1}
I20260620 18:44:51.122862 21841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:51.205415 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21718
I20260620 18:44:51.221194 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21358
I20260620 18:44:51.222343 21420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:51.324795 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21358
2026-06-20T18:44:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3904 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:44:51 dist-test-slave-zfp8 krb5kdc[21873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:44:51 dist-test-slave-zfp8 krb5kdc[21873](info): set up 2 sockets
Jun 20 18:44:51 dist-test-slave-zfp8 krb5kdc[21873](info): commencing operation
krb5kdc: starting...
W20260620 18:44:52.412177 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:44:52 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:44:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.254.keytab.
Entry for principal oryx/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.254.keytab.
I20260620 18:44:52.622156 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:45705
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.254.keytab
--principal=oryx/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:45705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:52.940137 21886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:52.940459 21886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:52.940629 21886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:52.947757 21886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:52.947896 21886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:52.948004 21886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:52.948112 21886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:52.948189 21886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:52.957794 21886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:45705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.254.keytab
--principal=oryx/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:45705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:52.959137 21886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:52.960590 21886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:52.972327 21894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:52.972219 21892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:52.972540 21891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:52.973582 21886 server_base.cc:1034] running on GCE node
I20260620 18:44:52.974355 21886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:52.975852 21886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:52.977051 21886 hybrid_clock.cc:648] HybridClock initialized: now 1781981092977005 us; error 38 us; skew 500 ppm
I20260620 18:44:52.977407 21886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:52 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:52.980661 21886 init.cc:377] Logged in from keytab as oryx/127.15.102.254@KRBTEST.COM (short username oryx)
I20260620 18:44:52.982095 21886 webserver.cc:460] Webserver started at http://127.15.102.254:42325/ using document root <none> and password file <none>
I20260620 18:44:52.982796 21886 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:52.982928 21886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:52.983278 21886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:52.986465 21886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "7df99ca7f44a4b128af3f42f9317c179"
format_stamp: "Formatted at 2026-06-20 18:44:52 on dist-test-slave-zfp8"
server_key: "fe5dd463e8a2546d4b62a33f49c83f2b"
server_key_iv: "158e9bce73361e95d680bd38724619e8"
server_key_version: "encryptionkey@0"
I20260620 18:44:52.987318 21886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "7df99ca7f44a4b128af3f42f9317c179"
format_stamp: "Formatted at 2026-06-20 18:44:52 on dist-test-slave-zfp8"
server_key: "fe5dd463e8a2546d4b62a33f49c83f2b"
server_key_iv: "158e9bce73361e95d680bd38724619e8"
server_key_version: "encryptionkey@0"
I20260620 18:44:52.992348 21886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:44:52.995641 21901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:52.996889 21886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:44:52.997079 21886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "7df99ca7f44a4b128af3f42f9317c179"
format_stamp: "Formatted at 2026-06-20 18:44:52 on dist-test-slave-zfp8"
server_key: "fe5dd463e8a2546d4b62a33f49c83f2b"
server_key_iv: "158e9bce73361e95d680bd38724619e8"
server_key_version: "encryptionkey@0"
I20260620 18:44:52.997246 21886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:53.011536 21886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:53.015604 21886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:53.015889 21886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:53.034168 21886 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:45705
I20260620 18:44:53.034198 21952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:45705 every 8 connection(s)
I20260620 18:44:53.035604 21886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:44:53.039513 21953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:53.042120 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21886
I20260620 18:44:53.042340 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:44:53.042985 15771 external_mini_cluster.cc:1428] Setting key d477fe49c2887e476148891563e21501
I20260620 18:44:53.052102 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: Bootstrap starting.
I20260620 18:44:53.055877 21953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:53.057215 21953 log.cc:824] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:44:53 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:44:53.060616 21953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: No bootstrap required, opened a new log
I20260620 18:44:53.067158 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:53.045047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.15.102.254:45705)
0620 18:44:53.045800 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:53.045812 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:53.045853 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:44:53.047845 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:53.047867 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:53.048273 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:53.049009 (+ 736us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:53.050576 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:53.051971 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:53.053713 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:53.054377 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:53.054530 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:53.058990 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:53.059000 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:53.059027 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:53.059067 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:53.062820 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:53.063498 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:53.063506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:53.063515 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:53.063725 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:53.064063 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:53.064068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:53.064072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:53.064753 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:53.065060 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:53.065364 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 18:44:53.065765 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":347,"threads_started":1}
I20260620 18:44:53.070887 21953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } }
I20260620 18:44:53.071201 21953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:53.071367 21953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df99ca7f44a4b128af3f42f9317c179, State: Initialized, Role: FOLLOWER
I20260620 18:44:53.071828 21953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } }
I20260620 18:44:53.072009 21953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:44:53.072104 21953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:44:53.072220 21953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:53.074273 21953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } }
I20260620 18:44:53.074829 21953 leader_election.cc:302] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df99ca7f44a4b128af3f42f9317c179; no voters:
I20260620 18:44:53.075418 21953 leader_election.cc:288] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:44:53.075543 21958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:53.076193 21958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 1 LEADER]: Becoming Leader. State: Replica: 7df99ca7f44a4b128af3f42f9317c179, State: Running, Role: LEADER
I20260620 18:44:53.076658 21958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } }
I20260620 18:44:53.077332 21953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:53.079559 21960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df99ca7f44a4b128af3f42f9317c179. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } }
I20260620 18:44:53.079504 21959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } }
I20260620 18:44:53.079921 21960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:53.079952 21959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:53.081881 21964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:53.087626 21964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:53.096091 21964 catalog_manager.cc:1347] Generated new cluster ID: 2f3ce4790e2c4e3ea6266e5aea720a32
I20260620 18:44:53.096216 21964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:53.110646 21964 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:44:53.112480 21964 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:53.121758 21964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: Generated new TSK 0
I20260620 18:44:53.122800 21964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab.
Entry for principal oryx/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab.
I20260620 18:44:53.172784 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45705
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab
--principal=oryx/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:53.498283 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:53.498572 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:53.498667 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:53.506181 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:53.506304 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:53.506420 21979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:53.516227 21979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab
--principal=oryx/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:53.517575 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:53.518919 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:53.531986 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:53.532155 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:53.532728 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:53.533219 21979 server_base.cc:1034] running on GCE node
I20260620 18:44:53.533775 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:53.534679 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:53.535866 21979 hybrid_clock.cc:648] HybridClock initialized: now 1781981093535804 us; error 51 us; skew 500 ppm
I20260620 18:44:53.536183 21979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:53 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981093, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:53.539309 21979 init.cc:377] Logged in from keytab as oryx/127.15.102.193@KRBTEST.COM (short username oryx)
I20260620 18:44:53.540781 21979 webserver.cc:460] Webserver started at http://127.15.102.193:41875/ using document root <none> and password file <none>
I20260620 18:44:53.541493 21979 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:53.541622 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:53.541939 21979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:53.544948 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "3a2463b6b32347518b053502d1eb9cae"
format_stamp: "Formatted at 2026-06-20 18:44:53 on dist-test-slave-zfp8"
server_key: "57ce40a487952102ddce3475ca36c383"
server_key_iv: "292a5933fb4d333d853246957fc174a1"
server_key_version: "encryptionkey@0"
I20260620 18:44:53.545760 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "3a2463b6b32347518b053502d1eb9cae"
format_stamp: "Formatted at 2026-06-20 18:44:53 on dist-test-slave-zfp8"
server_key: "57ce40a487952102ddce3475ca36c383"
server_key_iv: "292a5933fb4d333d853246957fc174a1"
server_key_version: "encryptionkey@0"
I20260620 18:44:53.551154 21979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:44:53.554788 21994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:53.556288 21979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:44:53.556471 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "3a2463b6b32347518b053502d1eb9cae"
format_stamp: "Formatted at 2026-06-20 18:44:53 on dist-test-slave-zfp8"
server_key: "57ce40a487952102ddce3475ca36c383"
server_key_iv: "292a5933fb4d333d853246957fc174a1"
server_key_version: "encryptionkey@0"
I20260620 18:44:53.556676 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:53.590762 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:53.594769 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:53.595072 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:53.596078 21979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:53.597399 21979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:53.597503 21979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:53.597653 21979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:53.597723 21979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:53.630548 21979 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:45793
I20260620 18:44:53.630596 22106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:45793 every 8 connection(s)
I20260620 18:44:53.631919 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:53.632545 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 21979
I20260620 18:44:53.632793 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:44:53.633486 15771 external_mini_cluster.cc:1428] Setting key 7de46a8eadbf0b28f7e41e5fe01ce9a9
Jun 20 18:44:53 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981093, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:44:53.656500 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:53.634723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:44983 (local address 127.15.102.254:45705)
0620 18:44:53.635211 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:53.635217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:53.636576 (+ 1359us) server_negotiation.cc:407] Connection header received
0620 18:44:53.639539 (+ 2963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:53.639547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:53.639638 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:53.639748 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:53.641846 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:53.642893 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:53.645578 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:53.645969 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:53.646075 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:53.650582 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:53.650587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:53.650590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:53.650622 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:53.653155 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:53.654164 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:53.654168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:53.654170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:53.654277 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:53.654792 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:53.654795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:53.654797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:53.655033 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:53.655137 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:53.656083 (+ 946us) server_negotiation.cc:299] Negotiation successful
0620 18:44:53.656249 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":240,"threads_started":1}
WARNING: no policy specified for oryx/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:44:53.657939 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:53.635539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.193:44983)
0620 18:44:53.636392 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:53.636455 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:53.639207 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:53.640026 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:53.640040 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:53.640836 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:53.641669 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:53.641696 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:53.643097 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:53.643105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:53.645395 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:53.645414 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:53.646209 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:53.646220 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:53.646547 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:53.647314 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:53.647353 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:53.650310 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:53.653303 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:53.653316 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:53.653337 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:53.654042 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:53.654388 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:53.654395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:53.654399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:53.654661 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:53.655158 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:53.655169 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:53.655658 (+ 489us) client_negotiation.cc:769] Sending connection context
0620 18:44:53.656001 (+ 343us) client_negotiation.cc:240] Negotiation successful
0620 18:44:53.656597 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":299,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.194@KRBTEST.COM" created.
I20260620 18:44:53.659718 22107 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45705
I20260620 18:44:53.660049 22107 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:53.660946 22107 heartbeater.cc:511] Master 127.15.102.254:45705 requested a full tablet report, sending...
I20260620 18:44:53.663112 21918 ts_manager.cc:194] Registered new tserver with Master: 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793)
I20260620 18:44:53.664651 21918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:44983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab.
Entry for principal oryx/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab.
I20260620 18:44:53.683720 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45705
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab
--principal=oryx/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:54.000981 22113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:54.001274 22113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:54.001367 22113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:54.008652 22113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:54.008785 22113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:54.008973 22113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:54.018800 22113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab
--principal=oryx/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:54.020174 22113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:54.021515 22113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:54.034852 22118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:54.034832 22119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:54.035933 22113 server_base.cc:1034] running on GCE node
W20260620 18:44:54.040086 22121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:54.040817 22113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:54.041795 22113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:54.043078 22113 hybrid_clock.cc:648] HybridClock initialized: now 1781981094043017 us; error 51 us; skew 500 ppm
I20260620 18:44:54.043478 22113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:54 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981094, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:54.046759 22113 init.cc:377] Logged in from keytab as oryx/127.15.102.194@KRBTEST.COM (short username oryx)
I20260620 18:44:54.048466 22113 webserver.cc:460] Webserver started at http://127.15.102.194:40035/ using document root <none> and password file <none>
I20260620 18:44:54.049165 22113 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:54.049305 22113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:54.049618 22113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:54.052556 22113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "d8810f82d15848ad930e74619a9d9f16"
format_stamp: "Formatted at 2026-06-20 18:44:54 on dist-test-slave-zfp8"
server_key: "aac2dca209d2d7909cb237f6db75dc2c"
server_key_iv: "9cf1165236e0ce495baac088176b21b8"
server_key_version: "encryptionkey@0"
I20260620 18:44:54.053507 22113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "d8810f82d15848ad930e74619a9d9f16"
format_stamp: "Formatted at 2026-06-20 18:44:54 on dist-test-slave-zfp8"
server_key: "aac2dca209d2d7909cb237f6db75dc2c"
server_key_iv: "9cf1165236e0ce495baac088176b21b8"
server_key_version: "encryptionkey@0"
I20260620 18:44:54.059108 22113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 18:44:54.062835 22128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:54.064276 22113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:44:54.064453 22113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "d8810f82d15848ad930e74619a9d9f16"
format_stamp: "Formatted at 2026-06-20 18:44:54 on dist-test-slave-zfp8"
server_key: "aac2dca209d2d7909cb237f6db75dc2c"
server_key_iv: "9cf1165236e0ce495baac088176b21b8"
server_key_version: "encryptionkey@0"
I20260620 18:44:54.064654 22113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:54.077065 22113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:54.081089 22113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:54.081385 22113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:54.082338 22113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:54.083835 22113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:54.083935 22113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:54.084085 22113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:54.084151 22113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:54.117332 22113 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:35487
I20260620 18:44:54.117377 22240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:35487 every 8 connection(s)
I20260620 18:44:54.118672 22113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:54.122951 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 22113
I20260620 18:44:54.123134 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:44:54.123620 15771 external_mini_cluster.cc:1428] Setting key 80e8f68823f8fdbab6981ddcf15ff606
Jun 20 18:44:54 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981094, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.195@KRBTEST.COM" created.
I20260620 18:44:54.145850 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:54.121205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:38705 (local address 127.15.102.254:45705)
0620 18:44:54.121390 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:54.121395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:54.122696 (+ 1301us) server_negotiation.cc:407] Connection header received
0620 18:44:54.124773 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:54.124777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:54.124840 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:54.124932 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:54.126549 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:54.127789 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:54.129286 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:54.130004 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:54.130162 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:54.139889 (+ 9727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:54.139895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:54.139898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:54.139932 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:54.142579 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:54.143413 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:54.143418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:54.143422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:54.143551 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:54.143997 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:54.144002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:54.144009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:54.144272 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:54.144419 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:54.145276 (+ 857us) server_negotiation.cc:299] Negotiation successful
0620 18:44:54.145502 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:44:54.146903 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:54.121726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.194:38705)
0620 18:44:54.122503 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:54.122579 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:54.124478 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:54.125182 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:54.125195 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:54.125847 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:54.126379 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:54.126395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:54.127924 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:54.127928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:54.129147 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:54.129158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:54.130211 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:54.130216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:54.130402 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:54.137052 (+ 6650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:54.137079 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:54.139630 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:54.142736 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:54.142746 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:54.142764 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:54.143291 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:54.143661 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:54.143666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:54.143669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:54.143876 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:54.144395 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:54.144404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:54.144809 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 18:44:54.145140 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 18:44:54.145727 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":327,"threads_started":1}
I20260620 18:44:54.148363 22241 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45705
I20260620 18:44:54.148721 22241 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:54.149654 22241 heartbeater.cc:511] Master 127.15.102.254:45705 requested a full tablet report, sending...
I20260620 18:44:54.151343 21918 ts_manager.cc:194] Registered new tserver with Master: d8810f82d15848ad930e74619a9d9f16 (127.15.102.194:35487)
I20260620 18:44:54.152154 21918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:38705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab.
Entry for principal oryx/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab.
I20260620 18:44:54.169883 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45705
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab
--principal=oryx/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:54.496455 22246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:54.496760 22246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:54.496855 22246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:54.504176 22246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:54.504292 22246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:54.504407 22246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:54.513834 22246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab
--principal=oryx/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:45705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:54.515091 22246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:54.516394 22246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:54.529137 22254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:54.529179 22252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:54.529892 22251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:54.530851 22246 server_base.cc:1034] running on GCE node
I20260620 18:44:54.531476 22246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:54.532348 22246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:54.533535 22246 hybrid_clock.cc:648] HybridClock initialized: now 1781981094533478 us; error 59 us; skew 500 ppm
I20260620 18:44:54.533856 22246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:54 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981094, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:54.536937 22246 init.cc:377] Logged in from keytab as oryx/127.15.102.195@KRBTEST.COM (short username oryx)
I20260620 18:44:54.538341 22246 webserver.cc:460] Webserver started at http://127.15.102.195:33687/ using document root <none> and password file <none>
I20260620 18:44:54.538980 22246 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:54.539079 22246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:54.539413 22246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:54.542398 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "766b84da71524a08b6dcd420e0ff44f8"
format_stamp: "Formatted at 2026-06-20 18:44:54 on dist-test-slave-zfp8"
server_key: "dd25781d2f046e24e8b7dd39c1a3bbf1"
server_key_iv: "e5b36aa8c9117f8aaeee55ac3e70c39c"
server_key_version: "encryptionkey@0"
I20260620 18:44:54.543552 22246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "766b84da71524a08b6dcd420e0ff44f8"
format_stamp: "Formatted at 2026-06-20 18:44:54 on dist-test-slave-zfp8"
server_key: "dd25781d2f046e24e8b7dd39c1a3bbf1"
server_key_iv: "e5b36aa8c9117f8aaeee55ac3e70c39c"
server_key_version: "encryptionkey@0"
I20260620 18:44:54.548841 22246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:44:54.552335 22261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:54.553730 22246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:44:54.553900 22246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "766b84da71524a08b6dcd420e0ff44f8"
format_stamp: "Formatted at 2026-06-20 18:44:54 on dist-test-slave-zfp8"
server_key: "dd25781d2f046e24e8b7dd39c1a3bbf1"
server_key_iv: "e5b36aa8c9117f8aaeee55ac3e70c39c"
server_key_version: "encryptionkey@0"
I20260620 18:44:54.554111 22246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:54.582450 22246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:54.587105 22246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:54.587427 22246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:54.588398 22246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:54.589773 22246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:54.589846 22246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:54.589968 22246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:54.590016 22246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:54.626520 22246 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:44391
I20260620 18:44:54.626616 22373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:44391 every 8 connection(s)
I20260620 18:44:54.627995 22246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:54.632073 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 22246
I20260620 18:44:54.632249 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:44:54.632838 15771 external_mini_cluster.cc:1428] Setting key f70f5237052e440ec29df713eb8991db
Jun 20 18:44:54 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981094, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:44:54.648706 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:54.630567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:44733 (local address 127.15.102.254:45705)
0620 18:44:54.630781 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:54.630787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:54.632106 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 18:44:54.634270 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:54.634275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:54.634403 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:54.634536 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:54.636559 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:54.637300 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:54.639001 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:54.639507 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:54.639616 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:54.642962 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:54.642966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:54.642970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:54.643001 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:54.645406 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:54.646103 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:54.646107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:54.646110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:54.646226 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:54.646721 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:54.646726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:54.646728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:54.647029 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:54.647145 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:54.648156 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0620 18:44:54.648369 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:44:54.649718 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:54.631069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.195:44733)
0620 18:44:54.631932 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:54.631986 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:54.633915 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:54.634926 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:54.634934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:54.635714 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:54.636409 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:54.636427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:54.637464 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:54.637469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:54.638845 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:54.638858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:54.639743 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:54.639748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:54.639945 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:54.640645 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:54.640667 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:54.642777 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:54.645567 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:54.645574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:54.645604 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:54.646001 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:54.646349 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:54.646354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:54.646357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:54.646616 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:54.647157 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:54.647166 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:54.647652 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 18:44:54.648041 (+ 389us) client_negotiation.cc:240] Negotiation successful
0620 18:44:54.648554 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":374,"threads_started":1}
I20260620 18:44:54.650969 22374 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45705
I20260620 18:44:54.651309 22374 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:54.652005 22374 heartbeater.cc:511] Master 127.15.102.254:45705 requested a full tablet report, sending...
I20260620 18:44:54.653339 21918 ts_manager.cc:194] Registered new tserver with Master: 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195:44391)
I20260620 18:44:54.654106 21918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:44733
I20260620 18:44:54.660609 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:44:54.667836 22107 heartbeater.cc:503] Master 127.15.102.254:45705 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.15.102.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.253.keytab.
Entry for principal oryx/127.15.102.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.253.keytab.
I20260620 18:44:54.704994 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.253:43553
--webserver_interface=127.15.102.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.253.keytab
--principal=oryx/127.15.102.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:45705,127.15.102.253:43553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:55.026664 22379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:55.026966 22379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:55.027055 22379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:55.034375 22379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:55.034556 22379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:55.034662 22379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:55.034757 22379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:55.034827 22379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:55.044965 22379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:45705,127.15.102.253:43553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.253.keytab
--principal=oryx/127.15.102.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.253:43553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.102.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
W20260620 18:44:55.045378 22379 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.102.254:45705,127.15.102.253:43553'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 18:44:55.046452 22379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:55.047837 22379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:55.059424 22384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:55.059368 22385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:55.059322 22387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:55.060616 22379 server_base.cc:1034] running on GCE node
I20260620 18:44:55.061326 22379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:55.062707 22379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:55.063925 22379 hybrid_clock.cc:648] HybridClock initialized: now 1781981095063866 us; error 33 us; skew 500 ppm
I20260620 18:44:55.064214 22379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:55.067459 22379 init.cc:377] Logged in from keytab as oryx/127.15.102.253@KRBTEST.COM (short username oryx)
I20260620 18:44:55.069188 22379 webserver.cc:460] Webserver started at http://127.15.102.253:35677/ using document root <none> and password file <none>
I20260620 18:44:55.069955 22379 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:55.070056 22379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:55.070331 22379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:55.074007 22379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data/instance:
uuid: "1872b7a8d9dc4fe6818078efe71ca39b"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "d536c50e6ed253f76fb1b5308fc229b0"
server_key_iv: "3fd3481f9d6b9e4e0e59cecc8e08390b"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.075114 22379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal/instance:
uuid: "1872b7a8d9dc4fe6818078efe71ca39b"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "d536c50e6ed253f76fb1b5308fc229b0"
server_key_iv: "3fd3481f9d6b9e4e0e59cecc8e08390b"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.080819 22379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:44:55.084455 22394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:55.085913 22379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:44:55.086074 22379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
uuid: "1872b7a8d9dc4fe6818078efe71ca39b"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "d536c50e6ed253f76fb1b5308fc229b0"
server_key_iv: "3fd3481f9d6b9e4e0e59cecc8e08390b"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.086274 22379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:55.120821 22379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:55.125587 22379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:55.125888 22379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:55.145727 22379 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.253:43553
I20260620 18:44:55.145745 22445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.253:43553 every 8 connection(s)
I20260620 18:44:55.147451 22379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
I20260620 18:44:55.151399 22446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:55.154227 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 22379
I20260620 18:44:55.154634 22241 heartbeater.cc:503] Master 127.15.102.254:45705 was elected leader, sending a full tablet report...
I20260620 18:44:55.159282 22446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.253@KRBTEST.COM
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.253@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:44:55.177984 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.162268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.15.102.254:45705)
0620 18:44:55.162747 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.162755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.163224 (+ 469us) server_negotiation.cc:407] Connection header received
0620 18:44:55.163445 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.163451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.163518 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.163627 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.164970 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.165912 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.167923 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.168402 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.168503 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.172124 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.172128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.172131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.172162 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.174886 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.175792 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.175798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.175807 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.175934 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.176354 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.176359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.176367 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.176677 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.176839 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.177375 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.177618 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":260,"threads_started":1}
I20260620 18:44:55.178738 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.156511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.15.102.253:43553)
0620 18:44:55.157353 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.157365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.157404 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:44:55.159150 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.159173 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.159669 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.159996 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.161832 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.163817 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.165438 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.166365 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.166494 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.171456 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.171465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.171478 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.171508 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.174369 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.175054 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.175060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.175068 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.175221 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.175644 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.175650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.175654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.176197 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.176495 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.176947 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.177412 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":638,"thread_start_us":397,"threads_started":1}
I20260620 18:44:55.178781 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.162398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45148)
0620 18:44:55.163100 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.163134 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.163322 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.163686 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.163693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.164518 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.164814 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.164832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.166059 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.166063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.167762 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.167778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.168643 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.168653 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.168889 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.169588 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.169621 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.171933 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.175075 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.175087 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.175095 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.175655 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.176062 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.176068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.176073 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.176232 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.176813 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.176821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.177006 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.177255 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.177596 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":21,"thread_start_us":363,"threads_started":1}
I20260620 18:44:55.181416 22446 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.253@KRBTEST.COM for oryx/127.15.102.253@KRBTEST.COM
I20260620 18:44:55.194084 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.182092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.0.0.1:47594)
0620 18:44:55.182273 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.182290 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.182384 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.182628 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.182632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.182959 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.183094 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.183101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.184102 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.184105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.185216 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.185229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.185917 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.185922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.186092 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.186671 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.186689 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.188675 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.191825 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.191831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.191835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.192174 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.192513 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.192518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.192526 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.192636 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.193137 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.193142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.193275 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.193405 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.193692 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:44:55.194116 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.182107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.15.102.253:43553)
0620 18:44:55.182278 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.182283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.182333 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:44:55.182465 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.182469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.182518 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.182593 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.183228 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.183964 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.185348 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.185778 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.185883 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.188844 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.188848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.188852 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.188880 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.191698 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.192284 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.192291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.192295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.192406 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.192745 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.192749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.192753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.193019 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.193162 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.193487 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.193814 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":66}
I20260620 18:44:55.198894 22446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Bootstrap starting.
I20260620 18:44:55.202220 22446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:55.203182 22446 log.cc:824] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:55.205715 22446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: No bootstrap required, opened a new log
I20260620 18:44:55.212570 22446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } }
I20260620 18:44:55.212872 22446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:55.212942 22446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1872b7a8d9dc4fe6818078efe71ca39b, State: Initialized, Role: FOLLOWER
I20260620 18:44:55.213454 22446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } }
I20260620 18:44:55.214200 22453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } } }
I20260620 18:44:55.214524 22453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:55.215299 22446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:44:55.223898 22464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:44:55.224050 22464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:44:55.225728 22379 master_runner.cc:428] Detected that this master 1872b7a8d9dc4fe6818078efe71ca39b is joining an existing cluster
I20260620 18:44:55.225836 22379 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.102.253:43553
I20260620 18:44:55.246031 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.229630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.15.102.254:45705)
0620 18:44:55.229824 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.229832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.230508 (+ 676us) server_negotiation.cc:407] Connection header received
0620 18:44:55.230785 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.230792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.230890 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.230967 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.232830 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.233884 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.236382 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.237139 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.237247 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.239888 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.239896 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.239904 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.239947 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.242989 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.243908 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.243915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.243919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.244066 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.244456 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.244461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.244465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.244830 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.244988 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.245451 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.245675 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:44:55.246153 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.229632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45154)
0620 18:44:55.230402 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.230422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.230625 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.231206 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.231210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.232080 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.232669 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.232684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.234093 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.234101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.236214 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.236234 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.237547 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.237556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.237748 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.238321 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.238347 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.239680 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.243128 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.243134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.243138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.243763 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.244178 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.244185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.244191 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.244347 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.244958 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.244964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.245181 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.245328 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.245660 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":3,"thread_start_us":646,"threads_started":1}
I20260620 18:44:55.246500 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.229742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.0.0.1:47608)
0620 18:44:55.230938 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.230954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.231084 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.231434 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.231438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.232099 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.232638 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.232655 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.234096 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.234101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.235840 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.235854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.237075 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.237081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.237271 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.237973 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.237998 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.239157 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.243300 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.243305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.243309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.243882 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.244267 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.244278 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.244283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.244443 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.245066 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.245069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.245189 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.245561 (+ 372us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.245838 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":36,"spinlock_wait_cycles":21376,"thread_start_us":982,"threads_started":1}
I20260620 18:44:55.247205 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.230105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.15.102.253:43553)
0620 18:44:55.230283 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.230289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.231041 (+ 752us) server_negotiation.cc:407] Connection header received
0620 18:44:55.231202 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.231207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.231293 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.231408 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.232790 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.233876 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.235985 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.236698 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.236790 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.239399 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.239406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.239411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.239449 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.243165 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.243996 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.244001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.244005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.244158 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.244540 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.244544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.244547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.244901 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.245023 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.245478 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.246858 (+ 1380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1156,"server-negotiator.queue_time_us":67}
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.254@KRBTEST.COM for oryx/127.15.102.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.102.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.252@KRBTEST.COM" created.
I20260620 18:44:55.267298 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.252501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.0.0.1:47616)
0620 18:44:55.253057 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.253120 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.253366 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.253846 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.253853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.254480 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.254692 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.254700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.255901 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.255904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.256871 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.256882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.257814 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.257819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.257962 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.258588 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.258634 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.261048 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.264334 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.264348 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.264356 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.264874 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.265240 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.265246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.265249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.265421 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.266096 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.266107 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.266313 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.266527 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.266849 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260620 18:44:55.271492 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.252577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.15.102.253:43553)
0620 18:44:55.252752 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.252760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.253216 (+ 456us) server_negotiation.cc:407] Connection header received
0620 18:44:55.253583 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.253588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.253685 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.253781 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.254836 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.255789 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.256997 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.257665 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.257781 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.261228 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.261233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.261237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.261271 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.264192 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.264989 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.264997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.265000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.265131 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.265557 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.265564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.265568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.265929 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.266082 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.266612 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.266848 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 18:44:55.273434 21918 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.102.253:43553: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b"
member_type: NON_VOTER
last_known_addr {
host: "127.15.102.253"
port: 43553
}
attrs {
promote: true
}
}
dest_uuid: "7df99ca7f44a4b128af3f42f9317c179"
cas_config_opid_index: -1
I20260620 18:44:55.274389 21918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } }
I20260620 18:44:55.289772 22477 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1872b7a8d9dc4fe6818078efe71ca39b (127.15.102.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } }
I20260620 18:44:55.290758 22421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:55.294068 22421 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Refusing update from remote peer 7df99ca7f44a4b128af3f42f9317c179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 18:44:55.294129 22477 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.102.253:43553
I20260620 18:44:55.294883 22379 master.cc:537] Master@127.15.102.253:43553 shutting down...
I20260620 18:44:55.295042 22477 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:55.294978 22476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } }
I20260620 18:44:55.296170 22476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.252.keytab.
Entry for principal oryx/127.15.102.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.252.keytab.
W20260620 18:44:55.299448 21902 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 18:44:55.300467 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.252:38521
--webserver_interface=127.15.102.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.252.keytab
--principal=oryx/127.15.102.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:45705,127.15.102.253:43553,127.15.102.252:38521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:44:55.304586 21902 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 -> Peer 1872b7a8d9dc4fe6818078efe71ca39b (127.15.102.253:43553): Couldn't send request to peer 1872b7a8d9dc4fe6818078efe71ca39b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 18:44:55.308979 22379 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:55.309393 22379 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:55.309588 22379 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: stopping tablet replica
I20260620 18:44:55.316205 22379 master.cc:552] Master@127.15.102.253:43553 shutdown complete.
I20260620 18:44:55.316298 22379 master_runner.cc:305] Clearing existing system tablet
I20260620 18:44:55.316673 22379 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:55.316781 22379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:55.319176 22379 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 18:44:55.320628 22482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:55.320911 22379 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:55.321175 22379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 18:44:55.321274 22379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
uuid: "1872b7a8d9dc4fe6818078efe71ca39b"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "d536c50e6ed253f76fb1b5308fc229b0"
server_key_iv: "3fd3481f9d6b9e4e0e59cecc8e08390b"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.324631 22379 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:55.329429 22379 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:44:55.329605 22379 log.cc:1196] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 18:44:55.330080 22379 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Deleting consensus metadata
I20260620 18:44:55.330328 22379 master_runner.cc:315] Copying system tablet from 127.15.102.254:45705
I20260620 18:44:55.336616 22379 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: tablet copy: Beginning tablet copy session from remote peer at address 127.15.102.254:45705
I20260620 18:44:55.355577 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.337383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.15.102.254:45705)
0620 18:44:55.337614 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.337623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.343589 (+ 5966us) server_negotiation.cc:407] Connection header received
0620 18:44:55.343815 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.343820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.343884 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.343990 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.344819 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.345536 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.347334 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.347974 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.348078 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.350385 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.350392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.350396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.350434 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.353033 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.353641 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.353648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.353652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.353778 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.354158 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.354163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.354166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.354430 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.354580 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.354952 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.355175 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:44:55.355662 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.337360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45170)
0620 18:44:55.343459 (+ 6099us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.343476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.343668 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.344014 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.344018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.344449 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.344673 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.344682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.345667 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.345674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.347122 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.347139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.348358 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.348365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.348564 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.349382 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.349408 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.350190 (+ 782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.353164 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.353171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.353175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.353519 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.353885 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.353891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.353894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.354035 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.354558 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.354563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.354717 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.354856 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.355154 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5961,"thread_start_us":5842,"threads_started":1}
I20260620 18:44:55.356676 21938 tablet_copy_service.cc:140] P 7df99ca7f44a4b128af3f42f9317c179: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1872b7a8d9dc4fe6818078efe71ca39b ({username='oryx', principal='oryx/127.15.102.253@KRBTEST.COM'} at 127.0.0.1:45170)
I20260620 18:44:55.356932 21938 tablet_copy_service.cc:161] P 7df99ca7f44a4b128af3f42f9317c179: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1872b7a8d9dc4fe6818078efe71ca39b at {username='oryx', principal='oryx/127.15.102.253@KRBTEST.COM'} at 127.0.0.1:45170: session id = 1872b7a8d9dc4fe6818078efe71ca39b-00000000000000000000000000000000
I20260620 18:44:55.360387 21938 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:44:55.362344 22379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:44:55.366909 22379 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: tablet copy: Starting download of 0 data blocks...
I20260620 18:44:55.367244 22379 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: tablet copy: Starting download of 1 WAL segments...
I20260620 18:44:55.369835 22379 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:44:55.373063 21938 tablet_copy_service.cc:342] P 7df99ca7f44a4b128af3f42f9317c179: Request end of tablet copy session 1872b7a8d9dc4fe6818078efe71ca39b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.102.253@KRBTEST.COM'} at 127.0.0.1:45170
I20260620 18:44:55.373248 21938 tablet_copy_service.cc:433] P 7df99ca7f44a4b128af3f42f9317c179: ending tablet copy session 1872b7a8d9dc4fe6818078efe71ca39b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1872b7a8d9dc4fe6818078efe71ca39b
I20260620 18:44:55.384361 22379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:55.387036 22489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:55.392675 22490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:55.397450 22379 server_base.cc:1034] running on GCE node
W20260620 18:44:55.397807 22492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:55.398289 22379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:55.399034 22379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:55.400274 22379 hybrid_clock.cc:648] HybridClock initialized: now 1781981095400195 us; error 33 us; skew 500 ppm
I20260620 18:44:55.400616 22379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:55.402956 22379 init.cc:377] Logged in from keytab as oryx/127.15.102.253@KRBTEST.COM (short username oryx)
I20260620 18:44:55.408142 22379 webserver.cc:460] Webserver started at http://127.15.102.253:39973/ using document root <none> and password file <none>
I20260620 18:44:55.408519 22379 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:55.408605 22379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:55.411772 22379 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 18:44:55.413651 22499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:55.414101 22379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 18:44:55.414258 22379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
uuid: "1872b7a8d9dc4fe6818078efe71ca39b"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "d536c50e6ed253f76fb1b5308fc229b0"
server_key_iv: "3fd3481f9d6b9e4e0e59cecc8e08390b"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.414425 22379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:55.440551 22379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:55.444538 22379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:55.467324 22379 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.253:43553
I20260620 18:44:55.467370 22550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.253:43553 every 8 connection(s)
I20260620 18:44:55.468452 22379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
I20260620 18:44:55.474094 22551 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:44:55.478382 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Bootstrap starting.
I20260620 18:44:55.495287 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:44:55.495808 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: Bootstrap complete.
I20260620 18:44:55.497089 22551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } }
I20260620 18:44:55.497422 22551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1872b7a8d9dc4fe6818078efe71ca39b, State: Initialized, Role: LEARNER
I20260620 18:44:55.497721 22551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } }
I20260620 18:44:55.498658 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } }
I20260620 18:44:55.499099 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: This master's current role is: LEARNER
I20260620 18:44:55.499167 22551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:55.513031 22565 catalog_manager.cc:1259] Loaded cluster ID: 2f3ce4790e2c4e3ea6266e5aea720a32
I20260620 18:44:55.513147 22565 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: loading cluster ID for follower catalog manager: success
I20260620 18:44:55.516721 22565 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: acquiring CA information for follower catalog manager: success
I20260620 18:44:55.519182 22565 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:44:55.656661 22374 heartbeater.cc:503] Master 127.15.102.254:45705 was elected leader, sending a full tablet report...
W20260620 18:44:55.692929 22480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:55.693226 22480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:55.693361 22480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:55.701618 22480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:44:55.701749 22480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:55.701864 22480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:55.701931 22480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:44:55.702010 22480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:44:55.712646 22480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:45705,127.15.102.253:43553,127.15.102.252:38521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.252.keytab
--principal=oryx/127.15.102.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.252:38521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.102.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:55.714042 22480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:55.715596 22480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:55.728082 22571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:55.728153 22573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:55.728060 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:55.729032 22480 server_base.cc:1034] running on GCE node
I20260620 18:44:55.729890 22480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:55.731534 22480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:55.732738 22480 hybrid_clock.cc:648] HybridClock initialized: now 1781981095732693 us; error 35 us; skew 500 ppm
I20260620 18:44:55.733053 22480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:55.736088 22480 init.cc:377] Logged in from keytab as oryx/127.15.102.252@KRBTEST.COM (short username oryx)
I20260620 18:44:55.737673 22480 webserver.cc:460] Webserver started at http://127.15.102.252:42155/ using document root <none> and password file <none>
I20260620 18:44:55.738372 22480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:55.738476 22480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:55.738823 22480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:44:55.741874 22480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data/instance:
uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "f9b0b9e9878eb95f45ad9b6f0b955625"
server_key_iv: "a8c8fb628a7f62bf845d07b089cb76df"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.742658 22480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal/instance:
uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "f9b0b9e9878eb95f45ad9b6f0b955625"
server_key_iv: "a8c8fb628a7f62bf845d07b089cb76df"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.748090 22480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:44:55.751549 22580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:55.752933 22480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:44:55.753114 22480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "f9b0b9e9878eb95f45ad9b6f0b955625"
server_key_iv: "a8c8fb628a7f62bf845d07b089cb76df"
server_key_version: "encryptionkey@0"
I20260620 18:44:55.753307 22480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:55.762555 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.755703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.0.0.1:47628)
0620 18:44:55.755875 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.755900 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.755998 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.757200 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.757205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.757235 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:55.757419 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.757428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.758755 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.758761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.760556 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.760569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.761750 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.761756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.761969 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.762028 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.762130 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.762277 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:44:55.762598 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.755899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.15.102.253:43553)
0620 18:44:55.756874 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.756878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.756897 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:44:55.756954 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.756957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.757023 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.757150 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:55.757567 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.758593 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.760702 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.761550 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.761764 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.761952 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.762154 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.762279 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":750,"threads_started":1}
I20260620 18:44:55.763689 22526 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 18:44:55.764235 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df99ca7f44a4b128af3f42f9317c179. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } }
I20260620 18:44:55.764483 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: This master's current role is: LEARNER
I20260620 18:44:55.768241 22480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:55.772178 22480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:55.772495 22480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:55.791440 22480 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.252:38521
I20260620 18:44:55.791457 22632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.252:38521 every 8 connection(s)
I20260620 18:44:55.792814 22480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
I20260620 18:44:55.796622 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 22480
I20260620 18:44:55.796885 22633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:55.805509 22633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.252@KRBTEST.COM
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.252@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:44:55.820655 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.798766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.15.102.252:38521)
0620 18:44:55.799569 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.799586 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.799636 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:44:55.801766 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.801794 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.802183 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.802580 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.804029 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.805284 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.806563 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.807202 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.807427 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.813059 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.813066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.813077 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.813112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.816564 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.817265 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.817277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.817283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.817433 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.817791 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.817798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.817801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.818444 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.818731 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.819194 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.819623 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":316,"threads_started":1}
I20260620 18:44:55.822839 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.808047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45184)
0620 18:44:55.808836 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.808876 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.809074 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.809499 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.809503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.810083 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.810718 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.810727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.811911 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.811918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.813409 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.813421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.814327 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.814331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.814518 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.815202 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.815218 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.817470 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.820432 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.820444 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.820451 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.821056 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.821395 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.821399 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.821402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.821558 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.822004 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.822012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.822171 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.822322 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.822553 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":406,"threads_started":1}
I20260620 18:44:55.822973 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.808077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.15.102.254:45705)
0620 18:44:55.808283 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.808287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.808965 (+ 678us) server_negotiation.cc:407] Connection header received
0620 18:44:55.809241 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.809247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.809341 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.809464 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.810863 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.811773 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.813542 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.814133 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.814262 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.817663 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.817673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.817677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.817710 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.820307 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.821167 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.821172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.821174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.821293 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.821654 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.821657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.821659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.821890 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.822039 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.822436 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.822652 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:44:55.824127 22633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.252@KRBTEST.COM for oryx/127.15.102.253@KRBTEST.COM
I20260620 18:44:55.836418 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.824799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.0.0.1:47634)
0620 18:44:55.824976 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.824993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.825084 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.825417 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.825421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.826183 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.826313 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.826321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.827300 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.827306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.828249 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.828260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.828953 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.828958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.829121 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.829684 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.829700 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.831697 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.834461 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.834464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.834467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.834823 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.835127 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.835130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.835132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.835326 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.835773 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.835776 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.835890 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.835986 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.836198 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:44:55.836489 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.824988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.15.102.253:43553)
0620 18:44:55.825167 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.825175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.825191 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:44:55.825243 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.825247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.825301 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.825401 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.826448 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.827150 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.828387 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.828798 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.828930 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.831890 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.831895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.831898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.831931 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.834340 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.834920 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.834924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.834927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.835040 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.835414 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.835417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.835420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.835682 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.835796 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.836077 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.836240 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:44:55.837450 22633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981095, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.252@KRBTEST.COM for oryx/127.15.102.252@KRBTEST.COM
I20260620 18:44:55.849479 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.838066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.0.0.1:33778)
0620 18:44:55.838229 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.838248 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.838325 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.838545 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.838548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.838843 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.838971 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.838978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.840053 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.840057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.840866 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.840875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.841521 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.841525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.841682 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.842235 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.842251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.844123 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.847343 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.847347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.847349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.847678 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.847986 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.847988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.847991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.848120 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.848596 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.848599 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.848784 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.848941 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.849176 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:44:55.849720 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.838128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.15.102.252:38521)
0620 18:44:55.838291 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.838295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.838309 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:44:55.838392 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.838395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.838445 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.838543 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.839111 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.839944 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.840980 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.841375 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.841498 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.844302 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.844307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.844310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.844338 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.847141 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.847777 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.847780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.847782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.847891 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.848214 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.848217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.848219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.848501 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.848614 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.849013 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.849318 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":76}
I20260620 18:44:55.854382 22633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Bootstrap starting.
I20260620 18:44:55.857415 22633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:55.858443 22633 log.cc:824] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:55.861203 22633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: No bootstrap required, opened a new log
I20260620 18:44:55.868441 22633 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } }
I20260620 18:44:55.868774 22633 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:55.868855 22633 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2aaefa93a814e8fb33e0f4dc2f6318c, State: Initialized, Role: FOLLOWER
I20260620 18:44:55.869335 22633 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } }
I20260620 18:44:55.870124 22639 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } } }
I20260620 18:44:55.870420 22639 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:55.871093 22633 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:44:55.880517 22650 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:44:55.880661 22650 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:44:55.884404 22480 master_runner.cc:428] Detected that this master a2aaefa93a814e8fb33e0f4dc2f6318c is joining an existing cluster
I20260620 18:44:55.884536 22480 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.102.252:38521
I20260620 18:44:55.922400 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.888773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45186)
0620 18:44:55.900930 (+ 12157us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.900961 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.901300 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.901971 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.901978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.902885 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.907830 (+ 4945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.907856 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.909896 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.909903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.911114 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.911125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.912134 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.912151 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.912339 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.913082 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.913373 (+ 291us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.914552 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.919197 (+ 4645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.919203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.919208 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.919723 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.920185 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.920191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.920195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.920397 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.921104 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.921109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.921391 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.921552 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.921881 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11975,"spinlock_wait_cycles":80640,"thread_start_us":11412,"threads_started":1}
I20260620 18:44:55.923171 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.889397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.15.102.253:43553)
0620 18:44:55.889867 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.889876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.908520 (+ 18644us) server_negotiation.cc:407] Connection header received
0620 18:44:55.908788 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.908794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.908867 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.908975 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.910258 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.911367 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.913150 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.913593 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.913716 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.916069 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.916077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.916084 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.916125 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.920034 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.920895 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.920901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.920904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.921054 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.921537 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.921543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.921546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.921883 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.921995 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.922632 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.922849 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313}
I20260620 18:44:55.923810 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.890032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.15.102.254:45705)
0620 18:44:55.890249 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.890257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.901082 (+ 10825us) server_negotiation.cc:407] Connection header received
0620 18:44:55.901481 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.901490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.901608 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.901696 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.908060 (+ 6364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.909733 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.911296 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.911932 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.912058 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.914743 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.914750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.914756 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.914798 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.919040 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.919887 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.919893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.919897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.920059 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.920545 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.920552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.920555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.920914 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.921042 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.922951 (+ 1909us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.923176 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:44:55.924265 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.888854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.0.0.1:47640)
0620 18:44:55.908406 (+ 19552us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.908430 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.908596 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.908993 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.908998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.909717 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.910083 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.910100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.911532 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.911538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.912985 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.913002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.913787 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.913793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.913962 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.914571 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.914595 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.915837 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.920185 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.920191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.920195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.920744 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.921197 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.921201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.921205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.921391 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.922095 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.922100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.922306 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.923453 (+ 1147us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.923773 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19394,"spinlock_wait_cycles":42880,"thread_start_us":18721,"threads_started":1}
I20260620 18:44:55.928166 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21979
I20260620 18:44:55.932442 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.889200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.15.102.252:38521)
0620 18:44:55.889677 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.889687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.908691 (+ 19004us) server_negotiation.cc:407] Connection header received
0620 18:44:55.915523 (+ 6832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.915528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.915590 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.915652 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.917099 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.918229 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.920259 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.921003 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.921095 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.923589 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.923595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.923600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.923633 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.929213 (+ 5580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.929967 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.929972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.929976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.930116 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.930824 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.930829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.930832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.931196 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.931381 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.931860 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.932069 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"spinlock_wait_cycles":33024}
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.193@KRBTEST.COM
Jun 20 18:44:55 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.254@KRBTEST.COM for oryx/127.15.102.252@KRBTEST.COM
I20260620 18:44:55.960534 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.888984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.0.0.1:33780)
0620 18:44:55.908534 (+ 19550us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.908552 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.915377 (+ 6825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.915888 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.915893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.916614 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.916951 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.916962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.918436 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.918441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.920094 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.920109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.921437 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.921444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.921633 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.922293 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.922315 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.923417 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.929367 (+ 5950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.929373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.929377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.929849 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.930267 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.930272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.930275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.930704 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.931445 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.931449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.931609 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.957853 (+ 26244us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.959661 (+ 1808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19372,"thread_start_us":19212,"threads_started":1}
I20260620 18:44:55.966938 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.930195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.15.102.252:38521)
0620 18:44:55.932730 (+ 2535us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.932738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.932759 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:44:55.932851 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.932856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.932958 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.933941 (+ 983us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.943289 (+ 9348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.944502 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.946432 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.947109 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.957671 (+ 10562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.959411 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.959418 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.959423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.959466 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.963700 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.964772 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.964778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.964782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.964912 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.965350 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.965360 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.965364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.965707 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.965845 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.966341 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.966551 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2437,"thread_start_us":15548,"threads_started":1}
I20260620 18:44:55.967180 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.929834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.15.102.193:45793)
0620 18:44:55.945217 (+ 15383us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:55.945231 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:55.945272 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:44:55.945362 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:55.945376 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:55.945723 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:55.946174 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:55.947563 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.948563 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.949822 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.950378 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.950510 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.958454 (+ 7944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:55.958483 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:55.958504 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:55.958547 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:55.963609 (+ 5062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.964460 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.964468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.964472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.964620 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:55.964996 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:55.965002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:55.965016 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:55.965339 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:55.965644 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:55.966567 (+ 923us) server_negotiation.cc:299] Negotiation successful
0620 18:44:55.966816 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15111,"thread_start_us":16032,"threads_started":1}
I20260620 18:44:55.967857 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:55.929546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.0.0.1:33784)
0620 18:44:55.929863 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:55.929881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:55.929974 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:55.933131 (+ 3157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:55.933136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:55.933591 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:55.933793 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.933806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.944754 (+ 10948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.944761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.946201 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:55.946222 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:55.955462 (+ 9240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:55.955468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:55.955648 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:55.956282 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:55.956306 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:55.959125 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:55.963828 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.963835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.963841 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.964315 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.965043 (+ 728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:55.965048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:55.965052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:55.965211 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:55.965842 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:55.965848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:55.966015 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 18:44:55.966218 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 18:44:55.966669 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260620 18:44:55.968642 22102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:55.969553 21918 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.102.252:38521: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c"
member_type: NON_VOTER
last_known_addr {
host: "127.15.102.252"
port: 38521
}
attrs {
promote: true
}
}
dest_uuid: "7df99ca7f44a4b128af3f42f9317c179"
cas_config_opid_index: 5
I20260620 18:44:55.970531 21918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } }
I20260620 18:44:55.972664 22476 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2aaefa93a814e8fb33e0f4dc2f6318c (127.15.102.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:55.976416 22526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 LEARNER]: Refusing update from remote peer 7df99ca7f44a4b128af3f42f9317c179: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 18:44:55.977140 22662 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 18:44:55.981189 22476 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.102.252:38521
I20260620 18:44:55.982762 22480 master.cc:537] Master@127.15.102.252:38521 shutting down...
I20260620 18:44:55.984331 22476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:55.985219 22476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:55.999300 22480 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:56.000551 22480 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:56.002300 22480 tablet_replica.cc:330] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: stopping tablet replica
I20260620 18:44:56.016987 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.016513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.0.0.1:33796)
0620 18:44:56.016690 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.016813 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:44:56.017347 21905 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 -> Peer a2aaefa93a814e8fb33e0f4dc2f6318c (127.15.102.252:38521): Couldn't send request to peer a2aaefa93a814e8fb33e0f4dc2f6318c. Status: Network error: Client connection negotiation failed: client connection to 127.15.102.252:38521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:44:56.019610 22526 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a2aaefa93a814e8fb33e0f4dc2f6318c (127.15.102.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.023092 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: NON_VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: true } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.023453 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: This master's current role is: LEARNER
I20260620 18:44:56.024134 22663 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: attempting to promote NON_VOTER 1872b7a8d9dc4fe6818078efe71ca39b to VOTER
I20260620 18:44:56.024964 22663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } }
I20260620 18:44:56.027280 22480 master.cc:552] Master@127.15.102.252:38521 shutdown complete.
I20260620 18:44:56.027362 22480 master_runner.cc:305] Clearing existing system tablet
I20260620 18:44:56.027561 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.026951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.0.0.1:33798)
0620 18:44:56.027228 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.027380 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 18:44:56.027778 22480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:56.027895 22480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:56.028956 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.028536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.0.0.1:33810)
0620 18:44:56.028727 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.028817 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 18:44:56.029155 21905 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 -> Peer a2aaefa93a814e8fb33e0f4dc2f6318c (127.15.102.252:38521): Couldn't send request to peer a2aaefa93a814e8fb33e0f4dc2f6318c. Status: Network error: Client connection negotiation failed: client connection to 127.15.102.252:38521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:44:56.031126 22526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 LEARNER]: Refusing update from remote peer 7df99ca7f44a4b128af3f42f9317c179: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 18:44:56.031805 22476 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 18:44:56.035573 22662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1872b7a8d9dc4fe6818078efe71ca39b (127.15.102.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.041101 22525 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1872b7a8d9dc4fe6818078efe71ca39b (127.15.102.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.041425 22662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.041884 22662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:56.042250 22476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.042562 22476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:56.045490 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.045748 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:56.047508 22480 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.010s sys 0.001s
I20260620 18:44:56.062685 22669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:56.063158 22480 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:56.063491 22480 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.006s sys 0.009s
I20260620 18:44:56.063613 22480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "f9b0b9e9878eb95f45ad9b6f0b955625"
server_key_iv: "a8c8fb628a7f62bf845d07b089cb76df"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.067582 22480 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:56.073446 22480 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 18:44:56.073662 22480 log.cc:1196] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 18:44:56.074141 22480 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Deleting consensus metadata
I20260620 18:44:56.074393 22480 master_runner.cc:315] Copying system tablet from 127.15.102.254:45705
I20260620 18:44:56.085306 22480 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.102.254:45705
I20260620 18:44:56.098161 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.086169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.15.102.254:45705)
0620 18:44:56.086486 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.086496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.087046 (+ 550us) server_negotiation.cc:407] Connection header received
0620 18:44:56.087280 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.087286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.087356 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.087458 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.088518 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.089323 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.090591 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.090984 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.091135 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.092952 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.092958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.092967 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.092998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.095675 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.096268 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.096278 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.096281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.096397 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.096764 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.096771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.096774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.097027 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.097175 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.097568 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.097780 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 18:44:56.098201 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.086132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45190)
0620 18:44:56.086929 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.086950 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.087121 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.087486 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.087490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.088150 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.088354 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.088363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.089462 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.089467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.090443 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.090454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.091155 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.091160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.091373 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.092034 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.092051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.092758 (+ 707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.095803 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.095809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.095814 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.096150 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.096500 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.096506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.096510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.096657 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.097154 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.097159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.097331 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.097471 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.097795 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":537,"threads_started":1}
I20260620 18:44:56.099663 21938 tablet_copy_service.cc:140] P 7df99ca7f44a4b128af3f42f9317c179: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a2aaefa93a814e8fb33e0f4dc2f6318c ({username='oryx', principal='oryx/127.15.102.252@KRBTEST.COM'} at 127.0.0.1:45190)
I20260620 18:44:56.099833 21938 tablet_copy_service.cc:161] P 7df99ca7f44a4b128af3f42f9317c179: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a2aaefa93a814e8fb33e0f4dc2f6318c at {username='oryx', principal='oryx/127.15.102.252@KRBTEST.COM'} at 127.0.0.1:45190: session id = a2aaefa93a814e8fb33e0f4dc2f6318c-00000000000000000000000000000000
I20260620 18:44:56.102211 21938 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 18:44:56.103787 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 18:44:56.108518 22480 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: tablet copy: Starting download of 0 data blocks...
I20260620 18:44:56.108798 22480 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: tablet copy: Starting download of 1 WAL segments...
I20260620 18:44:56.110826 22480 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 18:44:56.113833 21938 tablet_copy_service.cc:342] P 7df99ca7f44a4b128af3f42f9317c179: Request end of tablet copy session a2aaefa93a814e8fb33e0f4dc2f6318c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.102.252@KRBTEST.COM'} at 127.0.0.1:45190
I20260620 18:44:56.113958 21938 tablet_copy_service.cc:433] P 7df99ca7f44a4b128af3f42f9317c179: ending tablet copy session a2aaefa93a814e8fb33e0f4dc2f6318c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a2aaefa93a814e8fb33e0f4dc2f6318c
I20260620 18:44:56.125388 22480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:56.128021 22676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:56.128520 22677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:56.136040 22480 server_base.cc:1034] running on GCE node
W20260620 18:44:56.136204 22679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:56.136677 22480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:56.137390 22480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:56.138605 22480 hybrid_clock.cc:648] HybridClock initialized: now 1781981096138555 us; error 41 us; skew 500 ppm
I20260620 18:44:56.138850 22480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:56 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:56.140805 22480 init.cc:377] Logged in from keytab as oryx/127.15.102.252@KRBTEST.COM (short username oryx)
I20260620 18:44:56.144141 22480 webserver.cc:460] Webserver started at http://127.15.102.252:37257/ using document root <none> and password file <none>
I20260620 18:44:56.144451 22480 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:56.144559 22480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:56.146910 22480 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260620 18:44:56.148590 22686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:56.149489 22480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 18:44:56.149618 22480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c"
format_stamp: "Formatted at 2026-06-20 18:44:55 on dist-test-slave-zfp8"
server_key: "f9b0b9e9878eb95f45ad9b6f0b955625"
server_key_iv: "a8c8fb628a7f62bf845d07b089cb76df"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.149806 22480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:56.157629 22480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:56.161463 22480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:56.164531 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21979
I20260620 18:44:56.193243 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:45793
--local_ip_for_outbound_sockets=127.15.102.193
--tserver_master_addrs=127.15.102.254:45705
--webserver_port=41875
--webserver_interface=127.15.102.193
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab
--principal=oryx/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553,127.15.102.252:38521 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:44:56.202310 22480 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.252:38521
I20260620 18:44:56.203473 22480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
I20260620 18:44:56.202342 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.252:38521 every 8 connection(s)
I20260620 18:44:56.212101 22739 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:44:56.216511 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Bootstrap starting.
I20260620 18:44:56.235502 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:44:56.236027 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Bootstrap complete.
I20260620 18:44:56.237093 22739 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } }
I20260620 18:44:56.237378 22739 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a2aaefa93a814e8fb33e0f4dc2f6318c, State: Initialized, Role: LEARNER
I20260620 18:44:56.237643 22739 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } }
I20260620 18:44:56.238492 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.238955 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: This master's current role is: LEARNER
I20260620 18:44:56.238785 22739 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:44:56.256624 22753 catalog_manager.cc:1259] Loaded cluster ID: 2f3ce4790e2c4e3ea6266e5aea720a32
I20260620 18:44:56.256754 22753 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: loading cluster ID for follower catalog manager: success
I20260620 18:44:56.259305 22753 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: acquiring CA information for follower catalog manager: success
I20260620 18:44:56.261264 22753 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:44:56.449577 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.443599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.0.0.1:33826)
0620 18:44:56.443793 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.443810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.443913 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.444779 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.444784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.444810 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:56.444979 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.444987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.446211 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.446217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.447479 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.447492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.448772 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.448780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.448971 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.449006 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.449128 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.449296 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 18:44:56.449640 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.443781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.15.102.252:38521)
0620 18:44:56.444384 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.444388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.444403 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:44:56.444460 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.444464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.444527 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.444760 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:56.445119 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.446054 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.447637 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.448587 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.448798 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.448988 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.449143 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.449278 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":344,"threads_started":1}
I20260620 18:44:56.450850 22713 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 18:44:56.451690 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df99ca7f44a4b128af3f42f9317c179. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.452015 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: This master's current role is: LEARNER
W20260620 18:44:56.593868 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:56.594172 22736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:56.594329 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:56.602039 22736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:56.602207 22736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:56.602380 22736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:44:56.612308 22736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab
--principal=oryx/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:45793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=41875
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553,127.15.102.252:38521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:56.613714 22736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:56.615547 22736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:56.629594 22759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:56.629644 22760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:56.629801 22762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:56.631313 22736 server_base.cc:1034] running on GCE node
I20260620 18:44:56.632041 22736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:56.633112 22736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:56.634315 22736 hybrid_clock.cc:648] HybridClock initialized: now 1781981096634245 us; error 59 us; skew 500 ppm
I20260620 18:44:56.634642 22736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:56 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:56.637804 22736 init.cc:377] Logged in from keytab as oryx/127.15.102.193@KRBTEST.COM (short username oryx)
I20260620 18:44:56.639484 22736 webserver.cc:460] Webserver started at http://127.15.102.193:41875/ using document root <none> and password file <none>
I20260620 18:44:56.640316 22736 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:56.640460 22736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:56.646409 22736 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 18:44:56.649271 22769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:56.650681 22736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:56.650866 22736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "3a2463b6b32347518b053502d1eb9cae"
format_stamp: "Formatted at 2026-06-20 18:44:53 on dist-test-slave-zfp8"
server_key: "57ce40a487952102ddce3475ca36c383"
server_key_iv: "292a5933fb4d333d853246957fc174a1"
server_key_version: "encryptionkey@0"
I20260620 18:44:56.651965 22736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:56.671170 22736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:56.675199 22736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:56.675527 22736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:56.676481 22736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:56.677824 22736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:56.677904 22736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:56.678045 22736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:56.678140 22736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:56.717227 22736 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:45793
I20260620 18:44:56.717307 22881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:45793 every 8 connection(s)
I20260620 18:44:56.718636 22736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:44:56.724907 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 22736
I20260620 18:44:56.725318 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 22113
Jun 20 18:44:56 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.252@KRBTEST.COM
Jun 20 18:44:56 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
Jun 20 18:44:56 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981096, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.253@KRBTEST.COM
Jun 20 18:44:56 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.194@KRBTEST.COM
I20260620 18:44:56.748740 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.721598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45527 (local address 127.15.102.252:38521)
0620 18:44:56.723357 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.723363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.725337 (+ 1974us) server_negotiation.cc:407] Connection header received
0620 18:44:56.728574 (+ 3237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.728579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.728662 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.728729 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.731482 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.732560 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.734207 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.734863 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.734966 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.740877 (+ 5911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.740886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.740891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.740939 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.744600 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.745668 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.745674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.745677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.745815 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.746455 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.746460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.746464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.746822 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.746947 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.748108 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.748333 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1578}
I20260620 18:44:56.750048 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.722148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.15.102.193:45527)
0620 18:44:56.725154 (+ 3006us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.725226 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.728440 (+ 3214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.729118 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.729122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.730089 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.731262 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.731288 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.732690 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.732696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.734010 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.734025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.735417 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.735424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.735616 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.736325 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.736347 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.740585 (+ 4238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.744803 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.744814 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.744835 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.745512 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.746021 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.746024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.746026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.746320 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.747095 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.747105 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.747639 (+ 534us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.747982 (+ 343us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.748545 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2683,"mutex_wait_us":163,"spinlock_wait_cycles":11392,"thread_start_us":5134,"threads_started":1}
I20260620 18:44:56.750895 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.723557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:50069 (local address 127.15.102.253:43553)
0620 18:44:56.724067 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.724076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.727739 (+ 3663us) server_negotiation.cc:407] Connection header received
0620 18:44:56.728312 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.728317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.728402 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.728528 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.732733 (+ 4205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.733903 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.736716 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.737364 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.737487 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.743508 (+ 6021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.743515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.743521 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.743567 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.747658 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.748474 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.748480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.748484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.748613 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.749305 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.749310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.749313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.749637 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.749778 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.750284 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.750507 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":327,"threads_started":1}
I20260620 18:44:56.751793 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.722961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.15.102.193:50069)
0620 18:44:56.727612 (+ 4651us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.727641 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.728033 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.728980 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.728994 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.730089 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.731262 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.731288 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.734047 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.734054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.736565 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.736576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.738847 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.738854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.739102 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.739953 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.739994 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.743284 (+ 3290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.747824 (+ 4540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.747828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.747831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.748341 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.748761 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.748765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.748768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.749176 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.749853 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.749858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.750009 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.751069 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.751382 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4530,"mutex_wait_us":698,"spinlock_wait_cycles":4736,"thread_start_us":759,"threads_started":1}
I20260620 18:44:56.751929 22882 heartbeater.cc:347] Connected to a master server at 127.15.102.252:38521
I20260620 18:44:56.752233 22882 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:56.752846 22884 heartbeater.cc:347] Connected to a master server at 127.15.102.253:43553
I20260620 18:44:56.752997 22884 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:56.753144 22882 heartbeater.cc:511] Master 127.15.102.252:38521 requested a full tablet report, sending...
I20260620 18:44:56.753957 22884 heartbeater.cc:511] Master 127.15.102.253:43553 requested a full tablet report, sending...
I20260620 18:44:56.753994 22892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.726716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.15.102.194:35487)
0620 18:44:56.728147 (+ 1431us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.728154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.728175 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:44:56.728261 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.728269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.728896 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.729055 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.731117 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.734044 (+ 2927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.736453 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.737300 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.737430 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.744916 (+ 7486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.744933 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.744952 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.745004 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.749143 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.750385 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.750394 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.750397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.750578 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.751711 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.751716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.751719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.752145 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.752427 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.753334 (+ 907us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.753595 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":1170,"threads_started":1}
I20260620 18:44:56.754482 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.722174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.193:44809)
0620 18:44:56.725154 (+ 2980us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:56.725226 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:56.728342 (+ 3116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:56.728975 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:56.729002 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:56.730238 (+ 1236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:56.731272 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.731289 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.734088 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.734100 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.737687 (+ 3587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:56.737705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.738824 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.738833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:56.739102 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.740083 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:56.740107 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:56.743672 (+ 3565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:56.749344 (+ 5672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.749350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.749353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.749809 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.750480 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:56.750488 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:56.750494 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:56.750730 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:56.751641 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:56.751649 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:56.751802 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:44:56.752869 (+ 1067us) client_negotiation.cc:240] Negotiation successful
0620 18:44:56.753762 (+ 893us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2763,"mutex_wait_us":159,"spinlock_wait_cycles":164480,"thread_start_us":394,"threads_started":1}
I20260620 18:44:56.755537 22236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:56.755503 22703 ts_manager.cc:194] Registered new tserver with Master: 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793)
I20260620 18:44:56.756035 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:56.721527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:44809 (local address 127.15.102.254:45705)
0620 18:44:56.722641 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:56.722651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:56.725868 (+ 3217us) server_negotiation.cc:407] Connection header received
0620 18:44:56.728471 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:56.728480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:56.728576 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:56.728673 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:56.731430 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.732590 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.737832 (+ 5242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:56.738530 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:56.738685 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:56.743874 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:56.743880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:56.743885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:56.743925 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:56.747690 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.749927 (+ 2237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.749932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.749936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.750098 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:56.750830 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:56.750835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:56.750838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:56.751256 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:56.751407 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:56.755334 (+ 3927us) server_negotiation.cc:299] Negotiation successful
0620 18:44:56.755596 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":873,"threads_started":1}
I20260620 18:44:56.756822 22883 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45705
I20260620 18:44:56.756946 22883 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:56.757376 22883 heartbeater.cc:511] Master 127.15.102.254:45705 requested a full tablet report, sending...
I20260620 18:44:56.757583 22516 ts_manager.cc:194] Registered new tserver with Master: 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793)
I20260620 18:44:56.758551 21918 ts_manager.cc:194] Re-registered known tserver with Master: 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793)
I20260620 18:44:56.759467 21918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:44809
I20260620 18:44:56.845800 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 22113
I20260620 18:44:56.862680 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:35487
--local_ip_for_outbound_sockets=127.15.102.194
--tserver_master_addrs=127.15.102.254:45705
--webserver_port=40035
--webserver_interface=127.15.102.194
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab
--principal=oryx/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553,127.15.102.252:38521 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:44:56.943959 22667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: NON_VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: true } } }
I20260620 18:44:56.944387 22667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:56.945847 22896 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: attempting to promote NON_VOTER a2aaefa93a814e8fb33e0f4dc2f6318c to VOTER
I20260620 18:44:56.946444 22896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } }
I20260620 18:44:56.948248 22525 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Refusing update from remote peer 7df99ca7f44a4b128af3f42f9317c179: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:44:56.948275 22713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 LEARNER]: Refusing update from remote peer 7df99ca7f44a4b128af3f42f9317c179: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 18:44:56.948902 22667 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:44:56.949322 22895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 18:44:56.951753 22667 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2aaefa93a814e8fb33e0f4dc2f6318c (127.15.102.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:56.952291 22525 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2aaefa93a814e8fb33e0f4dc2f6318c (127.15.102.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:56.956037 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:56.956352 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:56.956597 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:56.956820 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:56.963759 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:56.964159 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:56.965852 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:56.966113 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179 [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:56.966460 22713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a2aaefa93a814e8fb33e0f4dc2f6318c (127.15.102.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:56.969592 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7df99ca7f44a4b128af3f42f9317c179" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:56.969901 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 18:44:57.195761 22894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:57.196044 22894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:57.196137 22894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:57.203715 22894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:57.203824 22894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:57.204001 22894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:44:57.213879 22894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab
--principal=oryx/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:35487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=40035
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553,127.15.102.252:38521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:57.215219 22894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:57.216668 22894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:57.229979 22905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:57.230011 22906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:57.230113 22908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:57.230511 22894 server_base.cc:1034] running on GCE node
I20260620 18:44:57.231168 22894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:57.232124 22894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:57.233315 22894 hybrid_clock.cc:648] HybridClock initialized: now 1781981097233279 us; error 36 us; skew 500 ppm
I20260620 18:44:57.233593 22894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981097, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:57.236675 22894 init.cc:377] Logged in from keytab as oryx/127.15.102.194@KRBTEST.COM (short username oryx)
I20260620 18:44:57.238082 22894 webserver.cc:460] Webserver started at http://127.15.102.194:40035/ using document root <none> and password file <none>
I20260620 18:44:57.238804 22894 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:57.238976 22894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:57.244740 22894 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:44:57.247614 22915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:57.248935 22894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:44:57.249102 22894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "d8810f82d15848ad930e74619a9d9f16"
format_stamp: "Formatted at 2026-06-20 18:44:54 on dist-test-slave-zfp8"
server_key: "aac2dca209d2d7909cb237f6db75dc2c"
server_key_iv: "9cf1165236e0ce495baac088176b21b8"
server_key_version: "encryptionkey@0"
I20260620 18:44:57.250066 22894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:57.274128 22894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:57.278661 22894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:57.278930 22894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:57.279980 22894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:57.281356 22894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:57.281481 22894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:57.281601 22894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:57.281682 22894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:57.313644 22894 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:35487
I20260620 18:44:57.313741 23027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:35487 every 8 connection(s)
I20260620 18:44:57.315368 22894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:44:57.322583 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 22894
I20260620 18:44:57.322983 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 22246
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.195@KRBTEST.COM
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981097, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981097, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.253@KRBTEST.COM
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981097, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.252@KRBTEST.COM
I20260620 18:44:57.371380 23039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.324308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.15.102.195:44391)
0620 18:44:57.341308 (+ 17000us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.341322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.344089 (+ 2767us) server_negotiation.cc:407] Connection header received
0620 18:44:57.344184 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.344195 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.344522 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.344703 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.347463 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.350018 (+ 2555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.352535 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.353411 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.353566 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.364330 (+ 10764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.364345 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.364362 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.364407 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.368192 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.369091 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.369101 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.369107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.369290 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.369691 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.369696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.369700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.370029 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.370341 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.370643 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.370910 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16742,"thread_start_us":16543,"threads_started":1}
I20260620 18:44:57.372752 22369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:57.375850 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.320960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:52459 (local address 127.15.102.253:43553)
0620 18:44:57.347201 (+ 26241us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.347208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.350927 (+ 3719us) server_negotiation.cc:407] Connection header received
0620 18:44:57.351349 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.351355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.351452 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.351538 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.357349 (+ 5811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.358922 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.360520 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.361100 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.361248 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.365791 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.365799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.365805 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.365865 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.370798 (+ 4933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.372490 (+ 1692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.372497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.372500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.372646 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.373545 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.373550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.373554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.373943 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.374103 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.375199 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.375446 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26101,"thread_start_us":27679,"threads_started":1}
I20260620 18:44:57.376392 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.318459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:34519 (local address 127.15.102.252:38521)
0620 18:44:57.349794 (+ 31335us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.349801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.350221 (+ 420us) server_negotiation.cc:407] Connection header received
0620 18:44:57.351731 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.351740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.351845 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.353674 (+ 1829us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.355112 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.356380 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.359809 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.360473 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.360590 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.368144 (+ 7554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.368153 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.368159 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.368204 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.372206 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.373073 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.373078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.373082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.373227 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.373792 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.373797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.373800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.374219 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.374374 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.375309 (+ 935us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.375602 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31159,"thread_start_us":31332,"threads_started":1}
I20260620 18:44:57.377197 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.319396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.15.102.194:34519)
0620 18:44:57.349763 (+ 30367us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:57.349789 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:57.351274 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:57.352044 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:57.352053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:57.353447 (+ 1394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:57.354888 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.354904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.356635 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.356644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.359624 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.359643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.360872 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.360881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.361179 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.361971 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:57.361993 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:57.367908 (+ 5915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:57.372372 (+ 4464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.372380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.372386 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.372924 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.373361 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.373369 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.373377 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.373555 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.374434 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:57.374439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:57.374687 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 18:44:57.375136 (+ 449us) client_negotiation.cc:240] Negotiation successful
0620 18:44:57.375687 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30141,"mutex_wait_us":775,"thread_start_us":33261,"threads_started":1}
I20260620 18:44:57.377609 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.320358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.15.102.194:52459)
0620 18:44:57.350823 (+ 30465us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:57.350843 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:57.351058 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:57.352034 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:57.352050 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:57.355680 (+ 3630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:57.357161 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.357175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.359098 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.359103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.360395 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.360410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.361487 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.361493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.361649 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.362203 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:57.362227 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:57.365571 (+ 3344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:57.371807 (+ 6236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.371812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.371816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.372339 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.372821 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.372828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.372833 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.373103 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.374200 (+ 1097us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:57.374209 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:57.374687 (+ 478us) client_negotiation.cc:769] Sending connection context
0620 18:44:57.376121 (+ 1434us) client_negotiation.cc:240] Negotiation successful
0620 18:44:57.376399 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30327,"mutex_wait_us":154,"spinlock_wait_cycles":19328,"thread_start_us":26896,"threads_started":1}
I20260620 18:44:57.378572 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.319396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.194:49601)
0620 18:44:57.344505 (+ 25109us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:57.344581 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:57.350954 (+ 6373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:57.352951 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:57.352957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:57.353746 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:57.354610 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.354638 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.356156 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.356161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.359606 (+ 3445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.359620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.360661 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.360667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.361032 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.361838 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:57.361882 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:57.365174 (+ 3292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:57.370996 (+ 5822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.371021 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.371040 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.371655 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.373215 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.373220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.373223 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.373386 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.377138 (+ 3752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:57.377146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:57.377352 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 18:44:57.377502 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 18:44:57.377803 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24724,"mutex_wait_us":135,"thread_start_us":29566,"threads_started":1}
I20260620 18:44:57.378995 23028 heartbeater.cc:347] Connected to a master server at 127.15.102.252:38521
I20260620 18:44:57.379032 23030 heartbeater.cc:347] Connected to a master server at 127.15.102.253:43553
I20260620 18:44:57.379380 23028 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:57.379385 23030 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:57.380203 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.318566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:49601 (local address 127.15.102.254:45705)
0620 18:44:57.349887 (+ 31321us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.349894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.349918 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:44:57.352128 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.352133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.352219 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.352338 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.354834 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.355967 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.359741 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.360426 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.360579 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.365409 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.365414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.365418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.365461 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.368981 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.371785 (+ 2804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.371791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.371794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.371940 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.376349 (+ 4409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.376354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.376360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.376753 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.376891 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.377660 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.377919 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31190,"thread_start_us":31555,"threads_started":1}
I20260620 18:44:57.380368 23030 heartbeater.cc:511] Master 127.15.102.253:43553 requested a full tablet report, sending...
I20260620 18:44:57.381012 23029 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45705
I20260620 18:44:57.381152 23029 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:57.381635 23029 heartbeater.cc:511] Master 127.15.102.254:45705 requested a full tablet report, sending...
I20260620 18:44:57.381762 23028 heartbeater.cc:511] Master 127.15.102.252:38521 requested a full tablet report, sending...
I20260620 18:44:57.382267 22516 ts_manager.cc:194] Registered new tserver with Master: d8810f82d15848ad930e74619a9d9f16 (127.15.102.194:35487)
I20260620 18:44:57.383498 21918 ts_manager.cc:194] Re-registered known tserver with Master: d8810f82d15848ad930e74619a9d9f16 (127.15.102.194:35487)
I20260620 18:44:57.383769 22703 ts_manager.cc:194] Registered new tserver with Master: d8810f82d15848ad930e74619a9d9f16 (127.15.102.194:35487)
I20260620 18:44:57.384590 21918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:49601
I20260620 18:44:57.462911 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 22246
I20260620 18:44:57.479859 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:44391
--local_ip_for_outbound_sockets=127.15.102.195
--tserver_master_addrs=127.15.102.254:45705
--webserver_port=33687
--webserver_interface=127.15.102.195
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab
--principal=oryx/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553,127.15.102.252:38521 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:44:57.761511 22883 heartbeater.cc:503] Master 127.15.102.254:45705 was elected leader, sending a full tablet report...
W20260620 18:44:57.807349 23041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:44:57.807699 23041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:44:57.807843 23041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:44:57.815968 23041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:44:57.816088 23041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:44:57.816264 23041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:44:57.826135 23041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab
--principal=oryx/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:44391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=33687
--tserver_master_addrs=127.15.102.254:45705,127.15.102.253:43553,127.15.102.252:38521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:44:57.827605 23041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:44:57.828944 23041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:44:57.842536 23050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:57.842607 23047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:44:57.842584 23048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:44:57.843691 23041 server_base.cc:1034] running on GCE node
I20260620 18:44:57.844358 23041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:44:57.845309 23041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:44:57.846498 23041 hybrid_clock.cc:648] HybridClock initialized: now 1781981097846417 us; error 68 us; skew 500 ppm
I20260620 18:44:57.846813 23041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981097, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:44:57.850106 23041 init.cc:377] Logged in from keytab as oryx/127.15.102.195@KRBTEST.COM (short username oryx)
I20260620 18:44:57.851655 23041 webserver.cc:460] Webserver started at http://127.15.102.195:33687/ using document root <none> and password file <none>
I20260620 18:44:57.852360 23041 fs_manager.cc:359] Metadata directory not provided
I20260620 18:44:57.852507 23041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:44:57.858261 23041 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:44:57.861179 23057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:57.862540 23041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:44:57.862723 23041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "766b84da71524a08b6dcd420e0ff44f8"
format_stamp: "Formatted at 2026-06-20 18:44:54 on dist-test-slave-zfp8"
server_key: "dd25781d2f046e24e8b7dd39c1a3bbf1"
server_key_iv: "e5b36aa8c9117f8aaeee55ac3e70c39c"
server_key_version: "encryptionkey@0"
I20260620 18:44:57.863759 23041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:44:57.877524 23041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:44:57.881960 23041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:44:57.882246 23041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:44:57.883181 23041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:44:57.884562 23041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:44:57.884680 23041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:57.884800 23041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:44:57.884881 23041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:44:57.917184 23041 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:44391
I20260620 18:44:57.917263 23169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:44391 every 8 connection(s)
I20260620 18:44:57.918608 23041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:44:57.920637 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 23041
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981097, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.253@KRBTEST.COM
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981097, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
Jun 20 18:44:57 dist-test-slave-zfp8 krb5kdc[21873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981097, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.252@KRBTEST.COM
I20260620 18:44:57.956966 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.928392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:40365 (local address 127.15.102.254:45705)
0620 18:44:57.934679 (+ 6287us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.934685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.934726 (+ 41us) server_negotiation.cc:407] Connection header received
0620 18:44:57.934776 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.934781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.934880 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.934990 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.936492 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.937532 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.939946 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.940855 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.940978 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.946211 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.946219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.946227 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.946294 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.950290 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.953124 (+ 2834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.953133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.953140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.953310 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.953895 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.953902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.953908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.954296 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.954447 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.956173 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.956458 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6109,"thread_start_us":6015,"threads_started":1}
I20260620 18:44:57.959208 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.927980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.195:40365)
0620 18:44:57.933938 (+ 5958us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:57.933958 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:57.934067 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:57.935147 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:57.935152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:57.935996 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:57.936344 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.936354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.937677 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.937683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.939739 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.939757 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.941345 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.941354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.941628 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.942256 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:57.942281 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:57.945570 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:57.952196 (+ 6626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.952209 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.952232 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.952953 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.953446 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.953454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.953460 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.953750 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.954523 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:57.954535 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:57.955043 (+ 508us) client_negotiation.cc:769] Sending connection context
0620 18:44:57.955404 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 18:44:57.957518 (+ 2114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5819,"mutex_wait_us":298,"thread_start_us":542,"threads_started":1}
I20260620 18:44:57.960369 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.922943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.15.102.195:37185)
0620 18:44:57.932415 (+ 9472us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:57.932439 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:57.932591 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:57.935114 (+ 2523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:57.935118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:57.935831 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:57.936181 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.936198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.937677 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.937683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.942171 (+ 4488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.942189 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.943457 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.943463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.943655 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.944366 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:57.944387 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:57.951102 (+ 6715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:57.955976 (+ 4874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.955984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.955990 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.956653 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.957645 (+ 992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.957652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.957659 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.957815 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.958981 (+ 1166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:57.958988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:57.959213 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 18:44:57.959441 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 18:44:57.959807 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9314,"mutex_wait_us":3,"thread_start_us":10667,"threads_started":1}
I20260620 18:44:57.961326 23171 heartbeater.cc:347] Connected to a master server at 127.15.102.254:45705
I20260620 18:44:57.961648 23171 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:57.962527 23171 heartbeater.cc:511] Master 127.15.102.254:45705 requested a full tablet report, sending...
I20260620 18:44:57.962568 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.921453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:37185 (local address 127.15.102.252:38521)
0620 18:44:57.934434 (+ 12981us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.934442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.934462 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:44:57.934717 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.934723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.934801 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.934937 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.936395 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.937491 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.942349 (+ 4858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.943020 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.943138 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.951429 (+ 8291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.951438 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.951447 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.951498 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.955747 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.957310 (+ 1563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.957318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.957324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.957495 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.958290 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.958297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.958302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.958762 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.958916 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.961813 (+ 2897us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.962100 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12850,"thread_start_us":12760,"threads_started":1}
I20260620 18:44:57.962664 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.922850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:45493 (local address 127.15.102.253:43553)
0620 18:44:57.927772 (+ 4922us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.927781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.928926 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 18:44:57.932398 (+ 3472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.932407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.932527 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.932619 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.935150 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.936228 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.939156 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.939867 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.939969 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.945537 (+ 5568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.945546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.945553 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.945612 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.952039 (+ 6427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.960386 (+ 8347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.960391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.960394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.960558 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.961009 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.961014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.961017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.961391 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.961520 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.962082 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.962302 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4770,"thread_start_us":6419,"threads_started":1}
I20260620 18:44:57.963584 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.922198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.15.102.195:45493)
0620 18:44:57.928776 (+ 6578us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:57.928822 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:57.932017 (+ 3195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:57.933147 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:57.933163 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:57.934081 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:44:57.934966 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.934994 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.936459 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.936467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.938995 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:57.939008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.940342 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.940349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:57.940640 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.941498 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:44:57.941537 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:44:57.945241 (+ 3704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:44:57.959782 (+ 14541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.959786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.959790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.960245 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.960717 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:44:57.960721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:44:57.960724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:44:57.960886 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:44:57.961617 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:44:57.961623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:44:57.961805 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 18:44:57.962834 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0620 18:44:57.963119 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6241,"mutex_wait_us":1,"thread_start_us":9272,"threads_started":1}
I20260620 18:44:57.965234 23172 heartbeater.cc:347] Connected to a master server at 127.15.102.253:43553
I20260620 18:44:57.965363 23172 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:57.965842 23172 heartbeater.cc:511] Master 127.15.102.253:43553 requested a full tablet report, sending...
I20260620 18:44:57.967087 22516 ts_manager.cc:194] Registered new tserver with Master: 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195:44391)
I20260620 18:44:57.968772 21918 ts_manager.cc:194] Re-registered known tserver with Master: 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195:44391)
I20260620 18:44:57.969812 21918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:40365
I20260620 18:44:57.974196 23170 heartbeater.cc:347] Connected to a master server at 127.15.102.252:38521
I20260620 18:44:57.974329 23170 heartbeater.cc:464] Registering TS with master...
I20260620 18:44:57.974694 23170 heartbeater.cc:511] Master 127.15.102.252:38521 requested a full tablet report, sending...
I20260620 18:44:57.975894 22703 ts_manager.cc:194] Registered new tserver with Master: 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195:44391)
I20260620 18:44:57.982124 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.939359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.15.102.254:45705)
0620 18:44:57.944183 (+ 4824us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.944190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.963600 (+ 19410us) server_negotiation.cc:407] Connection header received
0620 18:44:57.963857 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.963865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.963962 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.964088 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.966906 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.968055 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.970309 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.970914 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.971035 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.975185 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.975193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.975201 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.975268 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.979206 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.980036 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.980047 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.980050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.980167 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.980513 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.980521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.980526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.980801 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.981035 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.981564 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.981739 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4663,"thread_start_us":3922,"threads_started":1}
I20260620 18:44:57.986655 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.940619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.15.102.253:43553)
0620 18:44:57.961321 (+ 20702us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.961330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.968300 (+ 6970us) server_negotiation.cc:407] Connection header received
0620 18:44:57.968687 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.968695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.968840 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.968924 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.972309 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.973646 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.975589 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.976203 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.976297 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.979319 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.979329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.979336 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.979380 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.982834 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.984369 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.984373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.984375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.984531 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.985047 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.985052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.985055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.985377 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.985584 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.986034 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.986243 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20549,"thread_start_us":32977,"threads_started":1}
I20260620 18:44:57.990782 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:57.946357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.15.102.252:38521)
0620 18:44:57.962834 (+ 16477us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:57.962842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:57.973351 (+ 10509us) server_negotiation.cc:407] Connection header received
0620 18:44:57.973933 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:57.973940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:57.974030 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:57.974103 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:57.976964 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.978041 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.980006 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:57.980687 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:57.980828 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:57.984409 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:57.984417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:57.984423 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:57.984481 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:57.988340 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.988909 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.988915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.988918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.989069 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:57.989388 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:57.989393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:57.989397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:57.989736 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:57.989997 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:57.990200 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:44:57.990412 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16341,"thread_start_us":27123,"threads_started":1}
I20260620 18:44:57.986500 21918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:57.994931 21918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:58.037990 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.014818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.15.102.193:45793)
0620 18:44:58.027865 (+ 13047us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.027877 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.028189 (+ 312us) server_negotiation.cc:407] Connection header received
0620 18:44:58.028493 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.028505 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.028853 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.029103 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:58.029873 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.031714 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.035258 (+ 3544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.036954 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.037163 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.037343 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.037513 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.037680 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12831,"thread_start_us":13054,"threads_started":1}
I20260620 18:44:58.038074 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.014540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:45793 (local address 127.0.0.1:52386)
0620 18:44:58.028039 (+ 13499us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:58.028066 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:58.028279 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:58.029107 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:58.029113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:58.029148 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:58.029645 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.029662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.031940 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.031949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.035068 (+ 3119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.035079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.037170 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.037177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.037323 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.037356 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:44:58.037473 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:44:58.037603 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13314,"mutex_wait_us":23,"thread_start_us":18488,"threads_started":1}
I20260620 18:44:58.040740 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.016049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.15.102.194:35487)
0620 18:44:58.031825 (+ 15776us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.031837 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.032818 (+ 981us) server_negotiation.cc:407] Connection header received
0620 18:44:58.033001 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.033011 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.033288 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.033487 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:58.034344 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.035886 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.038286 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.039704 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.039869 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.040026 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.040261 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.040486 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15513,"thread_start_us":15382,"threads_started":1}
I20260620 18:44:58.040774 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.015012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35487 (local address 127.0.0.1:50240)
0620 18:44:58.032716 (+ 17704us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:58.032735 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:58.032860 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:58.033700 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:58.033703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:58.033721 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:58.034101 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.034111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.036033 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.036038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.038115 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.038128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.039949 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.039956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.040084 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.040106 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:44:58.040206 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:44:58.040328 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17563,"mutex_wait_us":1,"thread_start_us":17285,"threads_started":1}
I20260620 18:44:58.044974 22963 tablet_service.cc:1467] Processing CreateTablet for tablet a90f7f83434b4a5483bdaf72c931863b (DEFAULT_TABLE table=test-table [id=0e46df037f36487c8b72f2df40db013b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:58.046485 22963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90f7f83434b4a5483bdaf72c931863b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:58.046435 22817 tablet_service.cc:1467] Processing CreateTablet for tablet a90f7f83434b4a5483bdaf72c931863b (DEFAULT_TABLE table=test-table [id=0e46df037f36487c8b72f2df40db013b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:58.047761 22817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90f7f83434b4a5483bdaf72c931863b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:58.055042 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.015637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:44391 (local address 127.0.0.1:38702)
0620 18:44:58.033330 (+ 17693us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:58.033348 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:58.033475 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:58.044138 (+ 10663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:58.044142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:58.044170 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:58.044518 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.044528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.047545 (+ 3017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.047552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.049602 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.049618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.054239 (+ 4621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.054246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.054434 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.054482 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:44:58.054574 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:44:58.054716 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17545,"thread_start_us":12773,"threads_started":1}
I20260620 18:44:58.055316 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.016019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.15.102.195:44391)
0620 18:44:58.025939 (+ 9920us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.025949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.033473 (+ 7524us) server_negotiation.cc:407] Connection header received
0620 18:44:58.033608 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.033616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.033838 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.034046 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:58.044755 (+ 10709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.046426 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.049771 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.051755 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.052024 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.052594 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.054670 (+ 2076us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.054939 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9721,"thread_start_us":9723,"threads_started":1}
I20260620 18:44:58.059206 23105 tablet_service.cc:1467] Processing CreateTablet for tablet a90f7f83434b4a5483bdaf72c931863b (DEFAULT_TABLE table=test-table [id=0e46df037f36487c8b72f2df40db013b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:58.061113 23105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90f7f83434b4a5483bdaf72c931863b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:58.062147 23208 tablet_bootstrap.cc:492] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Bootstrap starting.
I20260620 18:44:58.063611 23207 tablet_bootstrap.cc:492] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: Bootstrap starting.
I20260620 18:44:58.065516 23208 tablet_bootstrap.cc:654] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:58.066819 23208 log.cc:824] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:58.066879 23207 tablet_bootstrap.cc:654] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:58.068429 23207 log.cc:824] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:58.070350 23208 tablet_bootstrap.cc:492] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: No bootstrap required, opened a new log
I20260620 18:44:58.070639 23208 ts_tablet_manager.cc:1397] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 18:44:58.071188 23207 tablet_bootstrap.cc:492] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: No bootstrap required, opened a new log
I20260620 18:44:58.071455 23207 ts_tablet_manager.cc:1397] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 18:44:58.078642 23207 raft_consensus.cc:348] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.079028 23207 raft_consensus.cc:374] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:58.079113 23207 raft_consensus.cc:729] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2463b6b32347518b053502d1eb9cae, State: Initialized, Role: FOLLOWER
I20260620 18:44:58.079710 23207 consensus_queue.cc:260] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.079834 23208 raft_consensus.cc:348] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.080245 23208 raft_consensus.cc:374] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:58.080340 23208 raft_consensus.cc:729] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8810f82d15848ad930e74619a9d9f16, State: Initialized, Role: FOLLOWER
I20260620 18:44:58.080966 23208 consensus_queue.cc:260] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.081512 23207 ts_tablet_manager.cc:1428] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260620 18:44:58.085381 23213 tablet_bootstrap.cc:492] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Bootstrap starting.
I20260620 18:44:58.087978 23208 ts_tablet_manager.cc:1428] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260620 18:44:58.088022 23029 heartbeater.cc:503] Master 127.15.102.254:45705 was elected leader, sending a full tablet report...
I20260620 18:44:58.091365 23213 tablet_bootstrap.cc:654] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:58.092600 23213 log.cc:824] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:44:58.095785 23213 tablet_bootstrap.cc:492] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: No bootstrap required, opened a new log
I20260620 18:44:58.096055 23213 ts_tablet_manager.cc:1397] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
W20260620 18:44:58.103528 23031 tablet.cc:2369] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:58.103988 23213 raft_consensus.cc:348] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.104472 23213 raft_consensus.cc:374] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:58.104574 23213 raft_consensus.cc:729] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766b84da71524a08b6dcd420e0ff44f8, State: Initialized, Role: FOLLOWER
I20260620 18:44:58.105144 23213 consensus_queue.cc:260] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.106094 23171 heartbeater.cc:503] Master 127.15.102.254:45705 was elected leader, sending a full tablet report...
I20260620 18:44:58.106762 23213 ts_tablet_manager.cc:1428] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 18:44:58.110986 23212 raft_consensus.cc:482] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:58.111213 23212 raft_consensus.cc:504] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.112834 23212 leader_election.cc:288] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793), 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195:44391)
I20260620 18:44:58.120008 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.113123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:38319 (local address 127.15.102.195:44391)
0620 18:44:58.113285 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.113290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.113905 (+ 615us) server_negotiation.cc:407] Connection header received
0620 18:44:58.114093 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.114100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.114171 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.114245 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:58.114978 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.116041 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.118272 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.119005 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.119189 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.119301 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.119637 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.119760 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:44:58.120018 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.113601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:49133 (local address 127.15.102.193:45793)
0620 18:44:58.113832 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.113839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.113972 (+ 133us) server_negotiation.cc:407] Connection header received
0620 18:44:58.114157 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.114160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.114215 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.114320 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:58.114913 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.115857 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.118214 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.118980 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.119168 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.119286 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.119634 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.119762 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:44:58.120190 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.113059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:44391 (local address 127.15.102.194:38319)
0620 18:44:58.113808 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:58.113828 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:58.113977 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:58.114296 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:58.114305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:58.114329 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:58.114773 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.114787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.116204 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.116210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.118123 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.118135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.119256 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.119263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.119447 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.119510 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 18:44:58.119613 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:44:58.119742 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":5,"thread_start_us":601,"threads_started":1}
I20260620 18:44:58.120245 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.113065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:45793 (local address 127.15.102.194:49133)
0620 18:44:58.113886 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:58.113906 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:58.114037 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:58.114350 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:58.114355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:58.114384 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:58.114745 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.114754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.115985 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.115991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.118055 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:58.118067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.119184 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.119190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:58.119463 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.119506 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 18:44:58.119608 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 18:44:58.119751 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":244,"thread_start_us":535,"threads_started":1}
I20260620 18:44:58.121181 23125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f7f83434b4a5483bdaf72c931863b" candidate_uuid: "d8810f82d15848ad930e74619a9d9f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "766b84da71524a08b6dcd420e0ff44f8" is_pre_election: true
I20260620 18:44:58.121447 22837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f7f83434b4a5483bdaf72c931863b" candidate_uuid: "d8810f82d15848ad930e74619a9d9f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2463b6b32347518b053502d1eb9cae" is_pre_election: true
I20260620 18:44:58.121760 23125 raft_consensus.cc:2454] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8810f82d15848ad930e74619a9d9f16 in term 0.
I20260620 18:44:58.121946 22837 raft_consensus.cc:2454] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8810f82d15848ad930e74619a9d9f16 in term 0.
I20260620 18:44:58.122483 22919 leader_election.cc:302] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a2463b6b32347518b053502d1eb9cae, 766b84da71524a08b6dcd420e0ff44f8, d8810f82d15848ad930e74619a9d9f16; no voters:
I20260620 18:44:58.122856 23212 raft_consensus.cc:2781] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:58.123018 23212 raft_consensus.cc:482] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:58.123131 23212 raft_consensus.cc:3037] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:58.125276 23212 raft_consensus.cc:504] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.125834 23212 leader_election.cc:288] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [CANDIDATE]: Term 1 election: Requested vote from peers 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793), 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195:44391)
I20260620 18:44:58.126238 22837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f7f83434b4a5483bdaf72c931863b" candidate_uuid: "d8810f82d15848ad930e74619a9d9f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2463b6b32347518b053502d1eb9cae"
I20260620 18:44:58.126331 23125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f7f83434b4a5483bdaf72c931863b" candidate_uuid: "d8810f82d15848ad930e74619a9d9f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "766b84da71524a08b6dcd420e0ff44f8"
I20260620 18:44:58.126426 22837 raft_consensus.cc:3037] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:58.126560 23125 raft_consensus.cc:3037] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:58.128772 22837 raft_consensus.cc:2454] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8810f82d15848ad930e74619a9d9f16 in term 1.
I20260620 18:44:58.128793 23125 raft_consensus.cc:2454] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8810f82d15848ad930e74619a9d9f16 in term 1.
I20260620 18:44:58.129190 22918 leader_election.cc:302] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2463b6b32347518b053502d1eb9cae, d8810f82d15848ad930e74619a9d9f16; no voters:
I20260620 18:44:58.129490 23212 raft_consensus.cc:2781] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:58.129909 23212 raft_consensus.cc:686] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 1 LEADER]: Becoming Leader. State: Replica: d8810f82d15848ad930e74619a9d9f16, State: Running, Role: LEADER
I20260620 18:44:58.130820 23212 consensus_queue.cc:237] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } }
I20260620 18:44:58.135793 21917 catalog_manager.cc:5495] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8810f82d15848ad930e74619a9d9f16 (127.15.102.194). New cstate: current_term: 1 leader_uuid: "d8810f82d15848ad930e74619a9d9f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } health_report { overall_health: UNKNOWN } } }
I20260620 18:44:58.171797 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.166852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.15.102.194:35487)
0620 18:44:58.167055 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.167068 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.167086 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:44:58.167310 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.167314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.167391 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.167515 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:58.167878 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.168629 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.169937 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.170325 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.170416 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.170822 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:58.171343 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:58.171447 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.171528 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.171596 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:44:58.176820 23173 tablet.cc:2369] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:44:58.180564 23125 raft_consensus.cc:1261] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 1 FOLLOWER]: Refusing update from remote peer d8810f82d15848ad930e74619a9d9f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:58.180568 22837 raft_consensus.cc:1261] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 1 FOLLOWER]: Refusing update from remote peer d8810f82d15848ad930e74619a9d9f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:58.181540 23218 consensus_queue.cc:1035] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:58.181862 23212 consensus_queue.cc:1035] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:58.195454 23221 mvcc.cc:205] Tried to move back new op lower bound from 7298994578137726976 to 7298994577947811840. Current Snapshot: MvccSnapshot[applied={T|T < 7298994578137726976}]
I20260620 18:44:58.217826 21918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:58.218146 21918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:58.224255 21918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7df99ca7f44a4b128af3f42f9317c179: Sending DeleteTablet for 3 replicas of tablet a90f7f83434b4a5483bdaf72c931863b
I20260620 18:44:58.226122 22817 tablet_service.cc:1514] Processing DeleteTablet for tablet a90f7f83434b4a5483bdaf72c931863b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:58 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:52386
I20260620 18:44:58.226971 23105 tablet_service.cc:1514] Processing DeleteTablet for tablet a90f7f83434b4a5483bdaf72c931863b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:58 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:38702
I20260620 18:44:58.227111 23231 tablet_replica.cc:330] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: stopping tablet replica
I20260620 18:44:58.228235 23231 raft_consensus.cc:2229] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:58.228740 23231 raft_consensus.cc:2258] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:58.228718 22963 tablet_service.cc:1514] Processing DeleteTablet for tablet a90f7f83434b4a5483bdaf72c931863b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:58 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:50240
I20260620 18:44:58.231897 23231 ts_tablet_manager.cc:1905] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:58.232362 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 21886
I20260620 18:44:58.234225 21948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:44:58.239691 23233 tablet_replica.cc:330] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: stopping tablet replica
I20260620 18:44:58.240360 23233 raft_consensus.cc:2229] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:58.241217 23233 raft_consensus.cc:2258] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:58.247279 23232 tablet_replica.cc:330] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: stopping tablet replica
I20260620 18:44:58.247460 23233 ts_tablet_manager.cc:1905] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:58.247954 23232 raft_consensus.cc:2229] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:58.248567 23232 raft_consensus.cc:2258] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:58.249183 23231 ts_tablet_manager.cc:1918] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:58.249372 23231 log.cc:1196] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/a90f7f83434b4a5483bdaf72c931863b
I20260620 18:44:58.249984 23231 ts_tablet_manager.cc:1939] T a90f7f83434b4a5483bdaf72c931863b P 3a2463b6b32347518b053502d1eb9cae: Deleting consensus metadata
I20260620 18:44:58.250919 23232 ts_tablet_manager.cc:1905] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:58.257705 23232 ts_tablet_manager.cc:1918] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:58.257958 23232 log.cc:1196] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/a90f7f83434b4a5483bdaf72c931863b
I20260620 18:44:58.258582 23232 ts_tablet_manager.cc:1939] T a90f7f83434b4a5483bdaf72c931863b P 766b84da71524a08b6dcd420e0ff44f8: Deleting consensus metadata
I20260620 18:44:58.258935 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.230362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.15.102.253:43553)
0620 18:44:58.230531 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.230552 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.230643 (+ 91us) server_negotiation.cc:407] Connection header received
0620 18:44:58.230971 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.230979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.231092 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.231183 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:58.240979 (+ 9796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.242116 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.246049 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.246726 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.246855 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.251321 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:58.251329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:58.251334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:58.251379 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:58.255478 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:58.256336 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:58.256342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:58.256346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:58.256494 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:58.256938 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:58.256944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:58.256948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:58.257357 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:58.257584 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.258311 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.258535 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:44:58.261256 23233 ts_tablet_manager.cc:1918] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:58.261437 23233 log.cc:1196] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/a90f7f83434b4a5483bdaf72c931863b
I20260620 18:44:58.261983 23233 ts_tablet_manager.cc:1939] T a90f7f83434b4a5483bdaf72c931863b P d8810f82d15848ad930e74619a9d9f16: Deleting consensus metadata
I20260620 18:44:58.277299 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.231585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.15.102.252:38521)
0620 18:44:58.231746 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.231768 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.260026 (+ 28258us) server_negotiation.cc:407] Connection header received
0620 18:44:58.260723 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.260729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.260810 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.260880 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:58.262823 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.264073 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.265919 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.266557 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.266712 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.270465 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:58.270472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:58.270477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:58.270522 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:58.274224 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:58.275011 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:58.275018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:58.275022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:58.275189 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:58.275676 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:58.275682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:58.275686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:58.276046 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:58.276294 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.276699 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.276918 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:44:58.388234 21905 catalog_manager.cc:4841] TS 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195:44391): tablet a90f7f83434b4a5483bdaf72c931863b (table test-table [id=0e46df037f36487c8b72f2df40db013b]) successfully deleted
I20260620 18:44:58.388329 21904 catalog_manager.cc:4841] TS 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793): tablet a90f7f83434b4a5483bdaf72c931863b (table test-table [id=0e46df037f36487c8b72f2df40db013b]) successfully deleted
I20260620 18:44:58.388265 21902 catalog_manager.cc:4841] TS d8810f82d15848ad930e74619a9d9f16 (127.15.102.194:35487): tablet a90f7f83434b4a5483bdaf72c931863b (table test-table [id=0e46df037f36487c8b72f2df40db013b]) successfully deleted
I20260620 18:44:58.388864 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 21886
I20260620 18:44:58.445363 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.417972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.15.102.253:43553)
0620 18:44:58.418169 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.418175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.428488 (+ 10313us) server_negotiation.cc:407] Connection header received
0620 18:44:58.428816 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.428823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.428905 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.428975 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:58.431425 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.432534 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.434546 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.435223 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.435405 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.439055 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:58.439064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:58.439070 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:58.439119 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:58.442485 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:58.443140 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:58.443148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:58.443151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:58.443330 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:58.443793 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:58.443798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:58.443802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:58.444151 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:58.444382 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.444801 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.445002 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:44:58.447137 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:58.417524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.15.102.252:38521)
0620 18:44:58.417805 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:58.417815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:58.429011 (+ 11196us) server_negotiation.cc:407] Connection header received
0620 18:44:58.429357 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:58.429365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:58.429475 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:58.429604 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:58.433515 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.434302 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.436345 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:58.436799 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:58.436969 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:58.440012 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:58.440018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:58.440023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:58.440065 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:58.443430 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:58.444140 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:58.444146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:58.444150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:58.444325 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:58.444773 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:58.444779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:58.444782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:58.445161 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:58.445378 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:58.446614 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0620 18:44:58.446833 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:44:59.204351 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.203676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.194:58573)
0620 18:44:59.204090 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.204201 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":207,"threads_started":1}
W20260620 18:44:59.204715 23029 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:45705 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
I20260620 18:44:59.215029 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.214115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.193:49501)
0620 18:44:59.214623 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.214812 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
W20260620 18:44:59.215670 22883 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:45705 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
I20260620 18:44:59.216545 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.215931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.195:53699)
0620 18:44:59.216292 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.216389 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":183,"threads_started":1}
W20260620 18:44:59.216850 23171 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:45705 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
I20260620 18:44:59.725548 23246 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7df99ca7f44a4b128af3f42f9317c179)
I20260620 18:44:59.725914 23247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7df99ca7f44a4b128af3f42f9317c179)
I20260620 18:44:59.725860 23246 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } }
I20260620 18:44:59.726192 23247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } }
I20260620 18:44:59.727605 23246 leader_election.cc:288] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df99ca7f44a4b128af3f42f9317c179 (127.15.102.254:45705), 1872b7a8d9dc4fe6818078efe71ca39b (127.15.102.253:43553)
I20260620 18:44:59.733863 23247 leader_election.cc:288] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7df99ca7f44a4b128af3f42f9317c179 (127.15.102.254:45705), a2aaefa93a814e8fb33e0f4dc2f6318c (127.15.102.252:38521)
I20260620 18:44:59.737680 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.727762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45664)
0620 18:44:59.737058 (+ 9296us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.737330 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9114,"thread_start_us":8940,"threads_started":1}
W20260620 18:44:59.741143 22688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
I20260620 18:44:59.742277 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.731223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45670)
0620 18:44:59.741732 (+ 10509us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.741921 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10009,"thread_start_us":10004,"threads_started":1}
W20260620 18:44:59.742635 22501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
I20260620 18:44:59.744359 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.743750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45682)
0620 18:44:59.743899 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.744113 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 18:44:59.744632 22688 leader_election.cc:334] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7df99ca7f44a4b128af3f42f9317c179 (127.15.102.254:45705): Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
I20260620 18:44:59.744870 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.744495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45690)
0620 18:44:59.744625 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.744719 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 18:44:59.745191 22501 leader_election.cc:334] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7df99ca7f44a4b128af3f42f9317c179 (127.15.102.254:45705): Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
I20260620 18:44:59.747162 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.727881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:43553 (local address 127.0.0.1:47692)
0620 18:44:59.737911 (+ 10030us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.737934 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:59.738095 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:59.740306 (+ 2211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:59.740311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:59.740338 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:59.740596 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.740609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.742671 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.742676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.745018 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.745032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.746364 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.746370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.746548 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.746571 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:44:59.746717 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 18:44:59.746859 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9928,"thread_start_us":14351,"threads_started":1}
I20260620 18:44:59.747189 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.728171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.15.102.253:43553)
0620 18:44:59.739931 (+ 11760us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.739940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.739965 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:44:59.740052 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.740060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.740159 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.740289 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:59.740854 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.742124 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.745164 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.746176 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.746372 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.746445 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.746741 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.746861 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":11574,"thread_start_us":11292,"threads_started":1}
I20260620 18:44:59.747851 22525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" is_pre_election: true
I20260620 18:44:59.748171 22525 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2aaefa93a814e8fb33e0f4dc2f6318c in term 1.
I20260620 18:44:59.748486 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.737081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.15.102.252:38521)
0620 18:44:59.741183 (+ 4102us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.741190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.741208 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:44:59.741276 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.741281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.741364 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.741463 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:59.742522 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.744038 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.746400 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.747510 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.747683 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.747755 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.748014 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.748127 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3935,"thread_start_us":3782,"threads_started":1}
I20260620 18:44:59.748543 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.727917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:38521 (local address 127.0.0.1:33862)
0620 18:44:59.740801 (+ 12884us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.740820 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:59.740926 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:59.741710 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:59.741715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:59.741744 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:59.742298 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.742315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.744204 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.744211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.746220 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.746233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.747704 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.747709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.747877 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.747897 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:44:59.747993 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:44:59.748111 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12756,"mutex_wait_us":21,"thread_start_us":12608,"threads_started":1}
I20260620 18:44:59.748730 22687 leader_election.cc:302] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1872b7a8d9dc4fe6818078efe71ca39b, a2aaefa93a814e8fb33e0f4dc2f6318c; no voters: 7df99ca7f44a4b128af3f42f9317c179
I20260620 18:44:59.749130 23246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 18:44:59.749081 22713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" is_pre_election: true
I20260620 18:44:59.749292 23246 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7df99ca7f44a4b128af3f42f9317c179)
I20260620 18:44:59.749418 23246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:44:59.751711 23246 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } }
I20260620 18:44:59.752136 22713 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1872b7a8d9dc4fe6818078efe71ca39b in current term 2: Already voted for candidate a2aaefa93a814e8fb33e0f4dc2f6318c in this term.
I20260620 18:44:59.752389 23246 leader_election.cc:288] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [CANDIDATE]: Term 2 election: Requested vote from peers 7df99ca7f44a4b128af3f42f9317c179 (127.15.102.254:45705), 1872b7a8d9dc4fe6818078efe71ca39b (127.15.102.253:43553)
I20260620 18:44:59.752929 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.752426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45694)
0620 18:44:59.752690 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.752786 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260620 18:44:59.752826 22525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1872b7a8d9dc4fe6818078efe71ca39b"
I20260620 18:44:59.753082 22525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:44:59.753211 22503 leader_election.cc:302] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1872b7a8d9dc4fe6818078efe71ca39b; no voters: 7df99ca7f44a4b128af3f42f9317c179, a2aaefa93a814e8fb33e0f4dc2f6318c
I20260620 18:44:59.753928 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.753500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45706)
0620 18:44:59.753718 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.753813 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 18:44:59.754077 22688 leader_election.cc:334] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7df99ca7f44a4b128af3f42f9317c179 (127.15.102.254:45705): Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
I20260620 18:44:59.755541 22525 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2aaefa93a814e8fb33e0f4dc2f6318c in term 2.
I20260620 18:44:59.755693 23247 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 18:44:59.755964 22687 leader_election.cc:302] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1872b7a8d9dc4fe6818078efe71ca39b, a2aaefa93a814e8fb33e0f4dc2f6318c; no voters: 7df99ca7f44a4b128af3f42f9317c179
I20260620 18:44:59.756227 23246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:44:59.756625 23246 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [term 2 LEADER]: Becoming Leader. State: Replica: a2aaefa93a814e8fb33e0f4dc2f6318c, State: Running, Role: LEADER
I20260620 18:44:59.757009 23246 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } }
I20260620 18:44:59.758754 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2aaefa93a814e8fb33e0f4dc2f6318c. Latest consensus state: current_term: 2 leader_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:59.759048 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:59.759696 23258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:44:59.763085 23258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:44:59.763914 23258 catalog_manager.cc:1259] Loaded cluster ID: 2f3ce4790e2c4e3ea6266e5aea720a32
I20260620 18:44:59.764014 23258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:44:59.765112 23258 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:44:59.765862 23258 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Loaded TSK: 0
I20260620 18:44:59.766664 23258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:44:59.803062 22702 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:44:59.809958 22702 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:44:59.813995 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.813582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45718)
0620 18:44:59.813755 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.813860 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:44:59.814108 22525 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [term 2 FOLLOWER]: Refusing update from remote peer a2aaefa93a814e8fb33e0f4dc2f6318c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 18:44:59.815256 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.814643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45732)
0620 18:44:59.814878 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.814982 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 18:44:59.815063 23256 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 18:44:59.815429 22688 consensus_peers.cc:487] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c -> Peer 7df99ca7f44a4b128af3f42f9317c179 (127.15.102.254:45705): Couldn't send request to peer 7df99ca7f44a4b128af3f42f9317c179. Status: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 18:44:59.817732 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2aaefa93a814e8fb33e0f4dc2f6318c. Latest consensus state: current_term: 2 leader_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:59.817986 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:59.819271 23256 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:59.819558 23256 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c [sys.catalog]: This master's current role is: LEADER
I20260620 18:44:59.819749 23260 mvcc.cc:205] Tried to move back new op lower bound from 7298994584832421888 to 7298994584608313344. Current Snapshot: MvccSnapshot[applied={T|T < 7298994584832421888}]
I20260620 18:44:59.820269 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7df99ca7f44a4b128af3f42f9317c179" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45705 } } peers { permanent_uuid: "1872b7a8d9dc4fe6818078efe71ca39b" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 43553 } attrs { promote: false } } peers { permanent_uuid: "a2aaefa93a814e8fb33e0f4dc2f6318c" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 38521 } attrs { promote: false } } }
I20260620 18:44:59.820508 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1872b7a8d9dc4fe6818078efe71ca39b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:44:59.836018 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.827150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35487 (local address 127.0.0.1:50270)
0620 18:44:59.827365 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.827385 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:59.827481 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:59.828689 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:59.828693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:59.828720 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:59.828894 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.828904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.831052 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.831060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.833698 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.833718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.835427 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.835433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.835624 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.835651 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 18:44:59.835703 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 18:44:59.835800 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 18:44:59.837286 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.827335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.15.102.194:35487)
0620 18:44:59.828310 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.828319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.828344 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:44:59.828430 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.828438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.828532 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.828658 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:59.829045 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.830885 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.833906 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.835261 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.836700 (+ 1439us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.836802 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.836850 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.836988 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":685,"threads_started":1}
I20260620 18:44:59.837944 22963 tablet_service.cc:1467] Processing CreateTablet for tablet 55b97241ed7643bca4d300f307342f35 (DEFAULT_TABLE table=test-table [id=221389f5919f4faa90d75a137a9c7dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:59.838408 22963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b97241ed7643bca4d300f307342f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:59.843309 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.828467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:45793 (local address 127.0.0.1:52402)
0620 18:44:59.836275 (+ 7808us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.836294 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:59.836413 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:59.836790 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:59.836795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:59.836820 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:59.836992 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.837001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.839096 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.839101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.841006 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.841020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.842540 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.842546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.842776 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.842809 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:44:59.842877 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 18:44:59.843005 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7702,"thread_start_us":13808,"threads_started":1}
I20260620 18:44:59.844240 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.828867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.15.102.193:45793)
0620 18:44:59.831418 (+ 2551us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.831423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.836360 (+ 4937us) server_negotiation.cc:407] Connection header received
0620 18:44:59.836543 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.836552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.836639 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.837123 (+ 484us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:59.837299 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.838899 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.841196 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.842337 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.843433 (+ 1096us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.843553 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.843620 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.843819 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2416,"thread_start_us":2279,"threads_started":1}
I20260620 18:44:59.845094 22817 tablet_service.cc:1467] Processing CreateTablet for tablet 55b97241ed7643bca4d300f307342f35 (DEFAULT_TABLE table=test-table [id=221389f5919f4faa90d75a137a9c7dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:59.845533 22817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b97241ed7643bca4d300f307342f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:59.846380 23269 tablet_bootstrap.cc:492] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: Bootstrap starting.
I20260620 18:44:59.848129 23269 tablet_bootstrap.cc:654] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:59.850307 23269 tablet_bootstrap.cc:492] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: No bootstrap required, opened a new log
I20260620 18:44:59.850445 23269 ts_tablet_manager.cc:1397] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 18:44:59.851279 23269 raft_consensus.cc:348] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.851518 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.827806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:44391 (local address 127.0.0.1:38706)
0620 18:44:59.827990 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.828015 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:59.828122 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:59.845435 (+ 17313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:59.845443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:59.845481 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:59.845857 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.845871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.847264 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.847271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.848604 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.848615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.850724 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.850730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.850949 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.850982 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:44:59.851048 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 18:44:59.851176 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 18:44:59.851517 23269 raft_consensus.cc:374] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:59.851605 23269 raft_consensus.cc:729] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8810f82d15848ad930e74619a9d9f16, State: Initialized, Role: FOLLOWER
I20260620 18:44:59.851862 23269 consensus_queue.cc:260] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.852197 23271 tablet_bootstrap.cc:492] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: Bootstrap starting.
I20260620 18:44:59.852449 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.827932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.15.102.195:44391)
0620 18:44:59.844968 (+ 17036us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.844974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.844992 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:44:59.845059 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.845063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.845168 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.845404 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:59.846073 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.847120 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.848753 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.850479 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.851403 (+ 924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.851780 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.851854 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.851992 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16919,"thread_start_us":17318,"threads_started":1}
I20260620 18:44:59.853174 23105 tablet_service.cc:1467] Processing CreateTablet for tablet 55b97241ed7643bca4d300f307342f35 (DEFAULT_TABLE table=test-table [id=221389f5919f4faa90d75a137a9c7dfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:44:59.853574 23105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b97241ed7643bca4d300f307342f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:44:59.854559 23271 tablet_bootstrap.cc:654] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:59.858985 23274 tablet_bootstrap.cc:492] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: Bootstrap starting.
I20260620 18:44:59.859289 23269 ts_tablet_manager.cc:1428] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: Time spent starting tablet: real 0.009s user 0.000s sys 0.009s
I20260620 18:44:59.861150 23274 tablet_bootstrap.cc:654] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: Neither blocks nor log segments found. Creating new log.
I20260620 18:44:59.862452 23271 tablet_bootstrap.cc:492] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: No bootstrap required, opened a new log
I20260620 18:44:59.862515 23028 heartbeater.cc:503] Master 127.15.102.252:38521 was elected leader, sending a full tablet report...
I20260620 18:44:59.862592 23271 ts_tablet_manager.cc:1397] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 18:44:59.863447 23271 raft_consensus.cc:348] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.863724 23271 raft_consensus.cc:374] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:59.863842 23271 raft_consensus.cc:729] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2463b6b32347518b053502d1eb9cae, State: Initialized, Role: FOLLOWER
I20260620 18:44:59.864141 23271 consensus_queue.cc:260] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.865443 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.859832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.194:35839)
0620 18:44:59.865102 (+ 5270us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.865254 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5136,"thread_start_us":5013,"threads_started":1}
I20260620 18:44:59.867033 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.866284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.194:43783)
0620 18:44:59.866508 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.866622 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 18:44:59.868088 23271 ts_tablet_manager.cc:1428] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260620 18:44:59.869091 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.868071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.193:40123)
0620 18:44:59.868675 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.868837 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":308,"threads_started":1}
I20260620 18:44:59.870460 23274 tablet_bootstrap.cc:492] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: No bootstrap required, opened a new log
I20260620 18:44:59.870482 22882 heartbeater.cc:503] Master 127.15.102.252:38521 was elected leader, sending a full tablet report...
I20260620 18:44:59.870481 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.870110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.193:39561)
0620 18:44:59.870247 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.870334 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:44:59.870605 23274 ts_tablet_manager.cc:1397] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260620 18:44:59.872128 23274 raft_consensus.cc:348] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.872355 23274 raft_consensus.cc:374] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:44:59.872444 23274 raft_consensus.cc:729] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766b84da71524a08b6dcd420e0ff44f8, State: Initialized, Role: FOLLOWER
I20260620 18:44:59.872670 23274 consensus_queue.cc:260] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.873436 23274 ts_tablet_manager.cc:1428] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 18:44:59.874753 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.873659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.195:54991)
0620 18:44:59.874463 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.874568 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":583,"threads_started":1}
I20260620 18:44:59.875264 23170 heartbeater.cc:503] Master 127.15.102.252:38521 was elected leader, sending a full tablet report...
I20260620 18:44:59.875787 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.875427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.195:43141)
0620 18:44:59.875563 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.875647 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 18:44:59.876955 23279 raft_consensus.cc:482] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:44:59.877112 23279 raft_consensus.cc:504] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.878489 23279 leader_election.cc:288] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8810f82d15848ad930e74619a9d9f16 (127.15.102.194:35487), 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793)
I20260620 18:44:59.883699 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.878679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35487 (local address 127.15.102.195:33413)
0620 18:44:59.878829 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.878847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:59.878985 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:59.879348 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:59.879352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:59.879374 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:59.879599 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.879608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.880521 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.880529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.881837 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.881849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.882934 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.882941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.883128 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.883160 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:44:59.883294 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:44:59.883438 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:44:59.883852 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.878766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:33413 (local address 127.15.102.194:35487)
0620 18:44:59.878949 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.878955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.878974 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:44:59.879106 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.879111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.879185 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.879333 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:59.879725 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.880401 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.881986 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.882749 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.882915 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.882990 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.883332 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.883533 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:44:59.884611 22983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b97241ed7643bca4d300f307342f35" candidate_uuid: "766b84da71524a08b6dcd420e0ff44f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8810f82d15848ad930e74619a9d9f16" is_pre_election: true
I20260620 18:44:59.885020 22983 raft_consensus.cc:2454] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 766b84da71524a08b6dcd420e0ff44f8 in term 0.
I20260620 18:44:59.885564 23058 leader_election.cc:302] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 766b84da71524a08b6dcd420e0ff44f8, d8810f82d15848ad930e74619a9d9f16; no voters:
I20260620 18:44:59.885957 23279 raft_consensus.cc:2781] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:44:59.886098 23279 raft_consensus.cc:482] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:44:59.886210 23279 raft_consensus.cc:3037] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:59.889034 23279 raft_consensus.cc:504] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.889496 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.878920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:44753 (local address 127.15.102.193:45793)
0620 18:44:59.879179 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.879187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.884001 (+ 4814us) server_negotiation.cc:407] Connection header received
0620 18:44:59.884246 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.884254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.884328 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.884393 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:44:59.884940 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.885772 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.887555 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.888757 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.888926 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.889007 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.889156 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.889281 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:44:59.889528 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.878678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:45793 (local address 127.15.102.195:44753)
0620 18:44:59.883922 (+ 5244us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.883940 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:44:59.884092 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:44:59.884574 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:44:59.884580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:44:59.884608 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:44:59.884776 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.884786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.885914 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.885920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.887421 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:44:59.887435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.888887 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.888891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:44:59.889021 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.889043 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 18:44:59.889112 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 18:44:59.889207 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5155,"mutex_wait_us":108,"thread_start_us":6109,"threads_started":1}
I20260620 18:44:59.889748 23279 leader_election.cc:288] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [CANDIDATE]: Term 1 election: Requested vote from peers d8810f82d15848ad930e74619a9d9f16 (127.15.102.194:35487), 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793)
I20260620 18:44:59.890205 22983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b97241ed7643bca4d300f307342f35" candidate_uuid: "766b84da71524a08b6dcd420e0ff44f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8810f82d15848ad930e74619a9d9f16"
I20260620 18:44:59.890424 22983 raft_consensus.cc:3037] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:59.890374 22837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b97241ed7643bca4d300f307342f35" candidate_uuid: "766b84da71524a08b6dcd420e0ff44f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2463b6b32347518b053502d1eb9cae" is_pre_election: true
I20260620 18:44:59.890516 22836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b97241ed7643bca4d300f307342f35" candidate_uuid: "766b84da71524a08b6dcd420e0ff44f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2463b6b32347518b053502d1eb9cae"
I20260620 18:44:59.890645 22837 raft_consensus.cc:2454] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 766b84da71524a08b6dcd420e0ff44f8 in term 0.
I20260620 18:44:59.892740 22983 raft_consensus.cc:2454] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 766b84da71524a08b6dcd420e0ff44f8 in term 1.
I20260620 18:44:59.893080 23058 leader_election.cc:302] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 766b84da71524a08b6dcd420e0ff44f8, d8810f82d15848ad930e74619a9d9f16; no voters: 3a2463b6b32347518b053502d1eb9cae
I20260620 18:44:59.893410 23279 raft_consensus.cc:2781] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:44:59.893832 23279 raft_consensus.cc:686] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 1 LEADER]: Becoming Leader. State: Replica: 766b84da71524a08b6dcd420e0ff44f8, State: Running, Role: LEADER
I20260620 18:44:59.894189 23279 consensus_queue.cc:237] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } }
I20260620 18:44:59.894999 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.894568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.195:37703)
0620 18:44:59.894725 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.894829 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 18:44:59.897768 22702 catalog_manager.cc:5495] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195). New cstate: current_term: 1 leader_uuid: "766b84da71524a08b6dcd420e0ff44f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "766b84da71524a08b6dcd420e0ff44f8" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 } health_report { overall_health: UNKNOWN } } }
I20260620 18:44:59.924324 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.918625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.15.102.195:44391)
0620 18:44:59.918848 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.918854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.918873 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:44:59.918964 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.918970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.919081 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.919219 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:44:59.919626 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.920725 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.921946 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.922600 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.922710 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.923047 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:44:59.923768 (+ 721us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:44:59.923921 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.923966 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.924042 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:44:59.928406 22836 raft_consensus.cc:3037] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:44:59.929008 22983 raft_consensus.cc:1261] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 1 FOLLOWER]: Refusing update from remote peer 766b84da71524a08b6dcd420e0ff44f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:59.929801 23279 consensus_queue.cc:1035] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8810f82d15848ad930e74619a9d9f16" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:59.931438 22836 raft_consensus.cc:1261] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [term 1 FOLLOWER]: Refusing update from remote peer 766b84da71524a08b6dcd420e0ff44f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:44:59.932103 23279 consensus_queue.cc:1035] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2463b6b32347518b053502d1eb9cae" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 45793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:44:59.932324 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.931688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.194:46459)
0620 18:44:59.931942 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.932105 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260620 18:44:59.935119 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.934659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.193:52845)
0620 18:44:59.934824 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.934933 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 18:44:59.937639 23275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.937093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.194:50781)
0620 18:44:59.937356 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.937460 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 18:44:59.941996 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.941457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.195:48367)
0620 18:44:59.941748 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.941845 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260620 18:44:59.947053 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.946231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.195:58281)
0620 18:44:59.946796 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.946882 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
I20260620 18:44:59.947705 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.946515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.15.102.193:46031)
0620 18:44:59.947467 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 18:44:59.947552 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":837}
I20260620 18:44:59.948725 23288 mvcc.cc:205] Tried to move back new op lower bound from 7298994585299808256 to 7298994585170522112. Current Snapshot: MvccSnapshot[applied={T|T < 7298994585299808256}]
I20260620 18:44:59.958632 22703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:59.958912 22703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:44:59.964097 22703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2aaefa93a814e8fb33e0f4dc2f6318c: Sending DeleteTablet for 3 replicas of tablet 55b97241ed7643bca4d300f307342f35
I20260620 18:44:59.964921 22963 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b97241ed7643bca4d300f307342f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:59 UTC) from {username='oryx', principal='oryx/127.15.102.252@KRBTEST.COM'} at 127.0.0.1:50270
I20260620 18:44:59.965183 23105 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b97241ed7643bca4d300f307342f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:59 UTC) from {username='oryx', principal='oryx/127.15.102.252@KRBTEST.COM'} at 127.0.0.1:38706
I20260620 18:44:59.966089 22817 tablet_service.cc:1514] Processing DeleteTablet for tablet 55b97241ed7643bca4d300f307342f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:44:59 UTC) from {username='oryx', principal='oryx/127.15.102.252@KRBTEST.COM'} at 127.0.0.1:52402
I20260620 18:44:59.967566 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 22736
I20260620 18:44:59.974472 23294 tablet_replica.cc:330] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: stopping tablet replica
I20260620 18:44:59.974825 23294 raft_consensus.cc:2229] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:59.975379 23294 raft_consensus.cc:2258] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:59.976976 23294 ts_tablet_manager.cc:1905] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:59.984068 23295 tablet_replica.cc:330] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: stopping tablet replica
I20260620 18:44:59.984294 23295 raft_consensus.cc:2229] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:44:59.984833 23296 tablet_replica.cc:330] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: stopping tablet replica
I20260620 18:44:59.984885 23295 raft_consensus.cc:2258] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:59.985060 23296 raft_consensus.cc:2229] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:44:59.985392 23296 raft_consensus.cc:2258] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:44:59.986615 23294 ts_tablet_manager.cc:1918] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:59.986757 23294 log.cc:1196] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/55b97241ed7643bca4d300f307342f35
I20260620 18:44:59.987223 23294 ts_tablet_manager.cc:1939] T 55b97241ed7643bca4d300f307342f35 P d8810f82d15848ad930e74619a9d9f16: Deleting consensus metadata
I20260620 18:44:59.987789 23295 ts_tablet_manager.cc:1905] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:59.988458 22687 catalog_manager.cc:4841] TS d8810f82d15848ad930e74619a9d9f16 (127.15.102.194:35487): tablet 55b97241ed7643bca4d300f307342f35 (table test-table [id=221389f5919f4faa90d75a137a9c7dfe]) successfully deleted
I20260620 18:44:59.989566 23296 ts_tablet_manager.cc:1905] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:44:59.995880 23295 ts_tablet_manager.cc:1918] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:59.996022 23295 log.cc:1196] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/55b97241ed7643bca4d300f307342f35
I20260620 18:44:59.996222 23296 ts_tablet_manager.cc:1918] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:44:59.996383 23296 log.cc:1196] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/55b97241ed7643bca4d300f307342f35
I20260620 18:44:59.996526 23295 ts_tablet_manager.cc:1939] T 55b97241ed7643bca4d300f307342f35 P 766b84da71524a08b6dcd420e0ff44f8: Deleting consensus metadata
I20260620 18:44:59.996850 23296 ts_tablet_manager.cc:1939] T 55b97241ed7643bca4d300f307342f35 P 3a2463b6b32347518b053502d1eb9cae: Deleting consensus metadata
I20260620 18:44:59.997721 22690 catalog_manager.cc:4841] TS 766b84da71524a08b6dcd420e0ff44f8 (127.15.102.195:44391): tablet 55b97241ed7643bca4d300f307342f35 (table test-table [id=221389f5919f4faa90d75a137a9c7dfe]) successfully deleted
I20260620 18:44:59.997928 22689 catalog_manager.cc:4841] TS 3a2463b6b32347518b053502d1eb9cae (127.15.102.193:45793): tablet 55b97241ed7643bca4d300f307342f35 (table test-table [id=221389f5919f4faa90d75a137a9c7dfe]) successfully deleted
I20260620 18:44:59.998210 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:44:59.968950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.15.102.193:45793)
0620 18:44:59.969113 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:44:59.969119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:44:59.974548 (+ 5429us) server_negotiation.cc:407] Connection header received
0620 18:44:59.975410 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:44:59.975417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:44:59.975499 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:44:59.975572 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:44:59.979222 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.980434 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.985286 (+ 4852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:44:59.986016 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:44:59.986150 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:44:59.990004 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:44:59.990022 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:44:59.990042 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:44:59.990085 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:44:59.994215 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:59.995059 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:59.995069 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:59.995072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:59.995211 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:44:59.995688 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:44:59.995693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:44:59.995697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:44:59.996355 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:44:59.996855 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0620 18:44:59.997476 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 18:44:59.997700 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:44:59.999549 22877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:00.093434 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 22736
I20260620 18:45:00.117211 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 22894
I20260620 18:45:00.130584 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:00.118088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.15.102.194:35487)
0620 18:45:00.118305 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:00.118311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:00.118331 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:45:00.118467 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:00.118477 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:00.118547 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:00.118647 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:00.120007 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:00.120734 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:00.121937 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:00.122322 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:00.122448 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:00.124934 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:00.124949 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:00.124965 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:00.124997 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:00.127922 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:00.128584 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:00.128592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:00.128594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:00.128721 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:00.129079 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:00.129086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:00.129089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:00.129445 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:00.129751 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:00.130032 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:45:00.130265 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:45:00.131682 23023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:00.261783 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 22894
I20260620 18:45:00.284004 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 23041
I20260620 18:45:00.300566 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:00.284894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.15.102.195:44391)
0620 18:45:00.285082 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:00.285087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:00.285219 (+ 132us) server_negotiation.cc:407] Connection header received
0620 18:45:00.285367 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:00.285372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:00.285441 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:00.285526 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:00.287443 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:00.288261 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:00.290069 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:00.290475 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:00.290563 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:00.294043 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:00.294058 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:00.294074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:00.294110 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:00.297608 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:00.298390 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:00.298401 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:00.298403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:00.298577 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:00.298965 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:00.298969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:00.298971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:00.299398 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:00.299721 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:00.300070 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 18:45:00.300233 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:45:00.301378 23165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:00.401666 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:00.400929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45748)
0620 18:45:00.401410 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:00.401493 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":282,"threads_started":1}
I20260620 18:45:00.402884 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:00.402559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:45705 (local address 127.0.0.1:45756)
0620 18:45:00.402694 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:00.402763 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 18:45:00.428211 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 23041
I20260620 18:45:00.449195 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 22379
I20260620 18:45:00.450282 22546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:00.591717 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 22379
I20260620 18:45:00.612215 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 22480
I20260620 18:45:00.613519 22733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:00.733035 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 22480
2026-06-20T18:45:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9411 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:45:00 dist-test-slave-zfp8 krb5kdc[23307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:45:00 dist-test-slave-zfp8 krb5kdc[23307](info): set up 2 sockets
Jun 20 18:45:00 dist-test-slave-zfp8 krb5kdc[23307](info): commencing operation
krb5kdc: starting...
W20260620 18:45:01.832333 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:45:01 dist-test-slave-zfp8 krb5kdc[23307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:45:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.254.keytab.
Entry for principal oryx/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.254.keytab.
I20260620 18:45:02.036746 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37979
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.254.keytab
--principal=oryx/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:37979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:02.356560 23320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:02.356871 23320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:02.356987 23320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:02.364148 23320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:02.364261 23320 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 18:45:02.364337 23320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:02.364434 23320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:02.364514 23320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:02.364614 23320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:02.376006 23320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:37979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.254.keytab
--principal=oryx/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:02.377333 23320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:02.378639 23320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:02.390487 23325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:02.390452 23326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:02.390452 23328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:02.391390 23320 server_base.cc:1034] running on GCE node
I20260620 18:45:02.392195 23320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:02.393536 23320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:02.394742 23320 hybrid_clock.cc:648] HybridClock initialized: now 1781981102394686 us; error 31 us; skew 500 ppm
I20260620 18:45:02.395073 23320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:02 dist-test-slave-zfp8 krb5kdc[23307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:02.398204 23320 init.cc:377] Logged in from keytab as oryx/127.15.102.254@KRBTEST.COM (short username oryx)
I20260620 18:45:02.399636 23320 webserver.cc:460] Webserver started at http://127.15.102.254:35781/ using document root <none> and password file <none>
I20260620 18:45:02.400339 23320 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:02.400447 23320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:02.400768 23320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:02.403724 23320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "601adf04640f4db98315eb65e3761506"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-zfp8"
server_key: "2dd18f52285ca9ded8c6a0ae8f2fe133"
server_key_iv: "e2bbb095f3e3d1d11764536228bed312"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.404505 23320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "601adf04640f4db98315eb65e3761506"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-zfp8"
server_key: "2dd18f52285ca9ded8c6a0ae8f2fe133"
server_key_iv: "e2bbb095f3e3d1d11764536228bed312"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.409380 23320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:45:02.412637 23335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:02.413862 23320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 18:45:02.414031 23320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "601adf04640f4db98315eb65e3761506"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-zfp8"
server_key: "2dd18f52285ca9ded8c6a0ae8f2fe133"
server_key_iv: "e2bbb095f3e3d1d11764536228bed312"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.414234 23320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:02.428534 23320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:02.432677 23320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:02.432946 23320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:02.459762 23320 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:37979
I20260620 18:45:02.459777 23396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:37979 every 8 connection(s)
I20260620 18:45:02.461512 23320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:02.465247 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:02.467362 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 23320
I20260620 18:45:02.467557 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:45:02.468168 15771 external_mini_cluster.cc:1428] Setting key 07fba578027683f4f2ec8a84a505cb19
I20260620 18:45:02.476203 23397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: Bootstrap starting.
I20260620 18:45:02.479995 23397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:02.481338 23397 log.cc:824] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: Log is configured to *not* fsync() on all Append() calls
Jun 20 18:45:02 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:45:02.484546 23397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: No bootstrap required, opened a new log
I20260620 18:45:02.491992 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:02.470153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.15.102.254:37979)
0620 18:45:02.470932 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:02.470945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:02.470983 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:45:02.472694 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:02.472716 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:02.473115 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:02.473536 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:02.474912 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:02.476247 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:02.478065 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:02.478710 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:02.478852 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:02.483695 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:02.483707 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:02.483728 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:02.483775 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:02.487647 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:02.488310 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:02.488319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:02.488328 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:02.488529 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:02.488844 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:02.488848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:02.488851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:02.489521 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:02.489802 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:02.490132 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 18:45:02.490578 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":356,"threads_started":1}
I20260620 18:45:02.495035 23397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601adf04640f4db98315eb65e3761506" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37979 } }
I20260620 18:45:02.495422 23397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:02.495501 23397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601adf04640f4db98315eb65e3761506, State: Initialized, Role: FOLLOWER
I20260620 18:45:02.496016 23397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601adf04640f4db98315eb65e3761506" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37979 } }
I20260620 18:45:02.496219 23397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:02.496315 23397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:02.496431 23397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:02.498704 23397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601adf04640f4db98315eb65e3761506" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37979 } }
I20260620 18:45:02.499137 23397 leader_election.cc:302] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 601adf04640f4db98315eb65e3761506; no voters:
I20260620 18:45:02.499754 23397 leader_election.cc:288] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:45:02.499919 23402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:02.500677 23402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [term 1 LEADER]: Becoming Leader. State: Replica: 601adf04640f4db98315eb65e3761506, State: Running, Role: LEADER
I20260620 18:45:02.501237 23402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601adf04640f4db98315eb65e3761506" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37979 } }
I20260620 18:45:02.501705 23397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:02.504283 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "601adf04640f4db98315eb65e3761506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601adf04640f4db98315eb65e3761506" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37979 } } }
I20260620 18:45:02.504312 23404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 601adf04640f4db98315eb65e3761506. Latest consensus state: current_term: 1 leader_uuid: "601adf04640f4db98315eb65e3761506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "601adf04640f4db98315eb65e3761506" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37979 } } }
I20260620 18:45:02.504787 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:02.504787 23404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:02.505671 23407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:02.511611 23407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:02.521009 23407 catalog_manager.cc:1347] Generated new cluster ID: db4ab006da8e4447817a0591df4caf48
I20260620 18:45:02.521123 23407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:02.532997 23407 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:45:02.534766 23407 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:02.541779 23407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: Generated new TSK 0
I20260620 18:45:02.542608 23407 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab.
Entry for principal oryx/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab.
I20260620 18:45:02.605032 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37979
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab
--principal=oryx/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:02.931080 23423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:02.931464 23423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:02.931636 23423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:02.938942 23423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:02.939088 23423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:02.939263 23423 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:45:02.939424 23423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:45:02.949340 23423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab
--principal=oryx/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:02.950668 23423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:02.952132 23423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:02.965165 23429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:02.965166 23428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:02.965502 23431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:02.965983 23423 server_base.cc:1034] running on GCE node
I20260620 18:45:02.966630 23423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:02.967624 23423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:02.968834 23423 hybrid_clock.cc:648] HybridClock initialized: now 1781981102968755 us; error 52 us; skew 500 ppm
I20260620 18:45:02.969120 23423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:02 dist-test-slave-zfp8 krb5kdc[23307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:02.972137 23423 init.cc:377] Logged in from keytab as oryx/127.15.102.193@KRBTEST.COM (short username oryx)
I20260620 18:45:02.973533 23423 webserver.cc:460] Webserver started at http://127.15.102.193:39731/ using document root <none> and password file <none>
I20260620 18:45:02.974224 23423 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:02.974341 23423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:02.974656 23423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:02.977532 23423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "b9bce56caa1f43e09c9ba1d857c1d24f"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-zfp8"
server_key: "ef269fafe7d711eee5a9b89cfa4ccd76"
server_key_iv: "e2224f86050d41426ae0b2988985890f"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.978345 23423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "b9bce56caa1f43e09c9ba1d857c1d24f"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-zfp8"
server_key: "ef269fafe7d711eee5a9b89cfa4ccd76"
server_key_iv: "e2224f86050d41426ae0b2988985890f"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.983568 23423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 18:45:02.987094 23438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:02.988428 23423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:02.988601 23423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "b9bce56caa1f43e09c9ba1d857c1d24f"
format_stamp: "Formatted at 2026-06-20 18:45:02 on dist-test-slave-zfp8"
server_key: "ef269fafe7d711eee5a9b89cfa4ccd76"
server_key_iv: "e2224f86050d41426ae0b2988985890f"
server_key_version: "encryptionkey@0"
I20260620 18:45:02.988843 23423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:03.009326 23423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:03.013707 23423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:03.013996 23423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:03.016789 23423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:03.016896 23423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.017019 23423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:03.017087 23423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:45:03 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:45:03.042091 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.017549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:52473 (local address 127.15.102.254:37979)
0620 18:45:03.018118 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:03.018122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:03.019434 (+ 1312us) server_negotiation.cc:407] Connection header received
0620 18:45:03.021551 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:03.021559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:03.021644 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:03.021773 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:03.024141 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.025231 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.027794 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.028396 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.028522 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.034081 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:03.034088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:03.034093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:03.034137 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:03.038134 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.039145 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.039150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.039154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.039336 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.039905 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.039910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.039914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.040330 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:03.040562 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:03.041454 (+ 892us) server_negotiation.cc:299] Negotiation successful
0620 18:45:03.041693 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":370,"threads_started":1}
I20260620 18:45:03.043514 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.018142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37979 (local address 127.15.102.193:52473)
0620 18:45:03.019225 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:03.019320 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:03.021238 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:03.022055 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:03.022071 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:03.022933 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:03.023873 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.023908 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.025395 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.025403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.027640 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.027654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.028778 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.028792 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.029128 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.030009 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:03.030044 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:03.033780 (+ 3736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:03.038303 (+ 4523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.038314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.038340 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.038991 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.039483 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.039491 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.039500 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.039771 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.040533 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:03.040542 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:03.041055 (+ 513us) client_negotiation.cc:769] Sending connection context
0620 18:45:03.041352 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 18:45:03.041922 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":390,"threads_started":1}
I20260620 18:45:03.063947 23447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.049689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:34685 (local address 127.15.102.254:37979)
0620 18:45:03.049883 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:03.049890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:03.050665 (+ 775us) server_negotiation.cc:407] Connection header received
0620 18:45:03.050945 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:03.050951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:03.051024 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:03.051128 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:03.052487 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.053434 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.055315 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.055718 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.055863 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.057695 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:03.057774 (+ 79us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:03.057789 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:03.057827 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:03.060805 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.061469 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.061476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.061481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.061647 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.062084 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.062091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.062095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.062465 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:03.062629 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:03.063416 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 18:45:03.063627 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:45:03.063975 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.049791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37979 (local address 127.15.102.193:34685)
0620 18:45:03.050537 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:03.050561 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:03.050739 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:03.051153 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:03.051157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:03.051834 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:03.052283 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.052299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.053671 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.053678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.055119 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.055132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.055925 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.055936 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.056094 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.056722 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:03.056739 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:03.057512 (+ 773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:03.060934 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.060941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.060945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.061339 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.061803 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.061807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.061811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.061936 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.062579 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:03.062584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:03.062903 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 18:45:03.063097 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 18:45:03.063494 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":379,"threads_started":1}
I20260620 18:45:03.071703 23423 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:36583
I20260620 18:45:03.071718 23558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:36583 every 8 connection(s)
I20260620 18:45:03.073091 23423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:03.075301 23559 heartbeater.cc:347] Connected to a master server at 127.15.102.254:37979
I20260620 18:45:03.075537 23559 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:03.076166 23559 heartbeater.cc:511] Master 127.15.102.254:37979 requested a full tablet report, sending...
I20260620 18:45:03.077986 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 23423
I20260620 18:45:03.078183 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:45:03.078483 23352 ts_manager.cc:194] Registered new tserver with Master: b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583)
I20260620 18:45:03.079078 15771 external_mini_cluster.cc:1428] Setting key c50cb585cdfd3bc4cf8392b6d066e75c
I20260620 18:45:03.080770 23352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:52473
WARNING: no policy specified for oryx/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab.
Entry for principal oryx/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab.
I20260620 18:45:03.126426 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37979
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab
--principal=oryx/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:03.454218 23563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:03.454502 23563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:03.454598 23563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:03.464218 23563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:03.464363 23563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:03.464499 23563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:45:03.464615 23563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:45:03.480211 23563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab
--principal=oryx/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:03.481798 23563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:03.483179 23563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:03.496440 23568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:03.496485 23571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:03.496520 23569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:03.497294 23563 server_base.cc:1034] running on GCE node
I20260620 18:45:03.497864 23563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:03.498780 23563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:03.499980 23563 hybrid_clock.cc:648] HybridClock initialized: now 1781981103499911 us; error 63 us; skew 500 ppm
I20260620 18:45:03.500242 23563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:03 dist-test-slave-zfp8 krb5kdc[23307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981103, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:03.503588 23563 init.cc:377] Logged in from keytab as oryx/127.15.102.194@KRBTEST.COM (short username oryx)
I20260620 18:45:03.505051 23563 webserver.cc:460] Webserver started at http://127.15.102.194:40847/ using document root <none> and password file <none>
I20260620 18:45:03.505790 23563 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:03.505916 23563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:03.506250 23563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:03.509159 23563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "606e725f5dbb4697bf8a1acae5b97617"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-zfp8"
server_key: "60600988a796b94d91b0ef5b4eac09b2"
server_key_iv: "5dc5ae659d0e2b162a36746497c45b7f"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.509960 23563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "606e725f5dbb4697bf8a1acae5b97617"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-zfp8"
server_key: "60600988a796b94d91b0ef5b4eac09b2"
server_key_iv: "5dc5ae659d0e2b162a36746497c45b7f"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.515043 23563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 18:45:03.518594 23578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.520084 23563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:03.520256 23563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "606e725f5dbb4697bf8a1acae5b97617"
format_stamp: "Formatted at 2026-06-20 18:45:03 on dist-test-slave-zfp8"
server_key: "60600988a796b94d91b0ef5b4eac09b2"
server_key_iv: "5dc5ae659d0e2b162a36746497c45b7f"
server_key_version: "encryptionkey@0"
I20260620 18:45:03.520479 23563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:03.561076 23563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:03.565076 23563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:03.565371 23563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:03.568153 23563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:03.568264 23563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:03.568431 23563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:03.568516 23563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:45:03 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981103, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:45:03.594444 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.568648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:37575 (local address 127.15.102.254:37979)
0620 18:45:03.569777 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:03.569785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:03.570632 (+ 847us) server_negotiation.cc:407] Connection header received
0620 18:45:03.572774 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:03.572780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:03.572887 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:03.573008 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:03.575965 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.577088 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.580216 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.580704 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.580853 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.586706 (+ 5853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:03.586712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:03.586715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:03.586747 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:03.589947 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.591200 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.591208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.591214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.591420 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.592154 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.592160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.592163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.592625 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:03.592782 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:03.593811 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0620 18:45:03.594066 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":876,"threads_started":1}
I20260620 18:45:03.595700 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.569530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37979 (local address 127.15.102.194:37575)
0620 18:45:03.570451 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:03.570504 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:03.572454 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:03.573448 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:03.573462 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:03.574635 (+ 1173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:03.575728 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.575766 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.577300 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.577309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.580063 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.580078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.580983 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.580993 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.581370 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.582191 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:03.582230 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:03.586367 (+ 4137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:03.590138 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.590148 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.590177 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.591017 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.591588 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.591592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.591594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.592010 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.592757 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:03.592766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:03.593336 (+ 570us) client_negotiation.cc:769] Sending connection context
0620 18:45:03.593704 (+ 368us) client_negotiation.cc:240] Negotiation successful
0620 18:45:03.594247 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":357,"threads_started":1}
I20260620 18:45:03.616878 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.602177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:46161 (local address 127.15.102.254:37979)
0620 18:45:03.602388 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:03.602397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:03.602978 (+ 581us) server_negotiation.cc:407] Connection header received
0620 18:45:03.603164 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:03.603170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:03.603292 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:03.603405 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:03.604613 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.605553 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.607602 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.608248 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.608412 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.610707 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:03.610712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:03.610715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:03.610746 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:03.614083 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.614866 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.614871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.614876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.614995 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:03.615445 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:03.615451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:03.615455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:03.615746 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:03.615935 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:03.616312 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 18:45:03.616530 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:45:03.617044 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:03.602217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37979 (local address 127.15.102.194:46161)
0620 18:45:03.602869 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:03.602897 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:03.603049 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:03.603427 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:03.603432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:03.603931 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:03.604467 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.604478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.605745 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.605752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.607450 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:03.607471 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:03.608473 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:03.608481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:03.608657 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:03.609357 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:03.609373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:03.610487 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:03.614209 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.614215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.614220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.614748 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.615115 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:03.615120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:03.615124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:03.615322 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:03.615888 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:03.615893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:03.616074 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 18:45:03.616219 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 18:45:03.616561 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":355,"threads_started":1}
I20260620 18:45:03.620386 23563 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:35443
I20260620 18:45:03.620473 23698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:35443 every 8 connection(s)
I20260620 18:45:03.621598 23563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:03.623387 23699 heartbeater.cc:347] Connected to a master server at 127.15.102.254:37979
I20260620 18:45:03.623616 23699 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:03.624255 23699 heartbeater.cc:511] Master 127.15.102.254:37979 requested a full tablet report, sending...
I20260620 18:45:03.625514 23352 ts_manager.cc:194] Registered new tserver with Master: 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443)
I20260620 18:45:03.626312 23352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:37575
I20260620 18:45:03.628059 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 23563
I20260620 18:45:03.628242 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:45:03.628767 15771 external_mini_cluster.cc:1428] Setting key 4a4a23a28dbc9367bb9ac57164862398
WARNING: no policy specified for oryx/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab.
Entry for principal oryx/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab.
I20260620 18:45:03.672315 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:37979
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab
--principal=oryx/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:03.991417 23703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:03.991712 23703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:03.991881 23703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:03.999006 23703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:03.999163 23703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:03.999357 23703 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 18:45:03.999517 23703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:45:04.009436 23703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab
--principal=oryx/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.102.254:37979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:04.010743 23703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:04.012102 23703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:04.025466 23711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:04.025444 23709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:04.025744 23708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:04.026237 23703 server_base.cc:1034] running on GCE node
I20260620 18:45:04.026813 23703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:04.027756 23703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:04.028982 23703 hybrid_clock.cc:648] HybridClock initialized: now 1781981104028921 us; error 31 us; skew 500 ppm
I20260620 18:45:04.029282 23703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:04 dist-test-slave-zfp8 krb5kdc[23307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981104, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:04.032526 23703 init.cc:377] Logged in from keytab as oryx/127.15.102.195@KRBTEST.COM (short username oryx)
I20260620 18:45:04.033910 23703 webserver.cc:460] Webserver started at http://127.15.102.195:42655/ using document root <none> and password file <none>
I20260620 18:45:04.034549 23703 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:04.034629 23703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:04.034893 23703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:04.037705 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "0aa02816cd3143e19c0ca41888ca44ed"
format_stamp: "Formatted at 2026-06-20 18:45:04 on dist-test-slave-zfp8"
server_key: "3bca9c3c84f279415aea4e3000dca67a"
server_key_iv: "66c84e17f0699e2fe7dbfcb430732d1c"
server_key_version: "encryptionkey@0"
I20260620 18:45:04.038434 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "0aa02816cd3143e19c0ca41888ca44ed"
format_stamp: "Formatted at 2026-06-20 18:45:04 on dist-test-slave-zfp8"
server_key: "3bca9c3c84f279415aea4e3000dca67a"
server_key_iv: "66c84e17f0699e2fe7dbfcb430732d1c"
server_key_version: "encryptionkey@0"
I20260620 18:45:04.043370 23703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 18:45:04.046638 23718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:04.047981 23703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:04.048149 23703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "0aa02816cd3143e19c0ca41888ca44ed"
format_stamp: "Formatted at 2026-06-20 18:45:04 on dist-test-slave-zfp8"
server_key: "3bca9c3c84f279415aea4e3000dca67a"
server_key_iv: "66c84e17f0699e2fe7dbfcb430732d1c"
server_key_version: "encryptionkey@0"
I20260620 18:45:04.048427 23703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:04.059996 23703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:04.064249 23703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:04.064509 23703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:04.066872 23703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:04.066947 23703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:04.067102 23703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:04.067176 23703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 18:45:04 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981104, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:45:04.084323 23559 heartbeater.cc:503] Master 127.15.102.254:37979 was elected leader, sending a full tablet report...
I20260620 18:45:04.091048 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.067767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:59481 (local address 127.15.102.254:37979)
0620 18:45:04.067927 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.067932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.069374 (+ 1442us) server_negotiation.cc:407] Connection header received
0620 18:45:04.071695 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.071703 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.071809 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.071889 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.074026 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.075307 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.077510 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.077917 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.078053 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.083030 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:04.083039 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:04.083044 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:04.083083 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:04.087432 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.088331 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.088336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.088340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.088462 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.088911 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.088917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.088922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.089229 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:04.089393 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.090484 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.090739 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 18:45:04.093109 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.068413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37979 (local address 127.15.102.195:59481)
0620 18:45:04.069216 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.069254 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.071318 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.072462 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.072485 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.073252 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:04.073861 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.073879 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.075509 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.075514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.077359 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.077371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.078143 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.078151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.078480 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.079409 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:04.079446 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:04.082759 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:04.087574 (+ 4815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:04.087580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:04.087594 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:04.088158 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:04.088573 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:04.088578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:04.088582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:04.088795 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:04.089389 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:04.089403 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:04.090011 (+ 608us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.090396 (+ 385us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.091120 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":370,"threads_started":1}
I20260620 18:45:04.114838 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.098818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:54349 (local address 127.15.102.254:37979)
0620 18:45:04.099026 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.099034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.099746 (+ 712us) server_negotiation.cc:407] Connection header received
0620 18:45:04.099935 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.099938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.099992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.100070 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.101484 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.102217 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.105022 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.105553 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.105662 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.109490 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:04.109496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:04.109501 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:04.109538 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:04.112251 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.112920 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.112928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.112931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.113088 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.113434 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.113439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.113442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.113709 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:04.113859 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.114276 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.114488 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:45:04.115013 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.098854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37979 (local address 127.15.102.195:54349)
0620 18:45:04.099617 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.099642 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.099820 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.100133 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.100141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.100861 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:04.101318 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.101334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.102351 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.102359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.104818 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.104834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.106005 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.106014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.106276 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.108495 (+ 2219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:04.108512 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:04.109292 (+ 780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:04.112398 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:04.112406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:04.112411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:04.112801 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:04.113197 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:04.113202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:04.113206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:04.113327 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:04.113836 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:04.113841 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:04.114039 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.114169 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.114495 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":383,"threads_started":1}
I20260620 18:45:04.130779 23703 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:36643
I20260620 18:45:04.130807 23837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:36643 every 8 connection(s)
I20260620 18:45:04.132053 23703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:04.134289 23838 heartbeater.cc:347] Connected to a master server at 127.15.102.254:37979
I20260620 18:45:04.134533 23838 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:04.134758 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 23703
I20260620 18:45:04.134919 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:45:04.135368 23838 heartbeater.cc:511] Master 127.15.102.254:37979 requested a full tablet report, sending...
I20260620 18:45:04.135470 15771 external_mini_cluster.cc:1428] Setting key 11e0b616aed8536b70c0641a2af68c50
I20260620 18:45:04.136833 23352 ts_manager.cc:194] Registered new tserver with Master: 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643)
I20260620 18:45:04.137794 23352 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:59481
I20260620 18:45:04.138720 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 18:45:04 dist-test-slave-zfp8 krb5kdc[23307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 18:45:04 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:04 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:45:04.186884 23587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:04.176787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.15.102.254:37979)
0620 18:45:04.177034 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.177041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.177277 (+ 236us) server_negotiation.cc:407] Connection header received
0620 18:45:04.177421 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.177425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.177487 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.177591 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.178900 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.179695 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.180889 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.181280 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.181375 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.186677 (+ 5302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34776: BlockingRecv error: recv got EOF from 127.0.0.1:34776 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 20 18:45:04 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:45:04.203433 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.189914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.15.102.254:37979)
0620 18:45:04.190123 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.190129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.190379 (+ 250us) server_negotiation.cc:407] Connection header received
0620 18:45:04.190506 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.190509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.190567 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.190653 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.191785 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.192616 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.193826 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.194273 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.194366 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.198189 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:04.198194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:04.198198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:04.198238 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:04.200807 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.201442 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.201453 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.201456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.201573 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.201952 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.201957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.201961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.202256 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:04.202522 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.202856 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.203078 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:45:04.207320 23351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:45:04.214314 23351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:04.241477 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.226782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.15.102.194:35443)
0620 18:45:04.233165 (+ 6383us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.233175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.233204 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:45:04.233318 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.233326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.233517 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.234611 (+ 1094us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.234756 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.236038 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.238803 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.240261 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.240462 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.240655 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.240981 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.241168 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6179,"thread_start_us":6035,"threads_started":1}
I20260620 18:45:04.241706 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.226593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35443 (local address 127.0.0.1:58446)
0620 18:45:04.228167 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.228185 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.228305 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.233722 (+ 5417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.233726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.233752 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.234408 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.234422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.236163 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.236171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.238641 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.238655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.240462 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.240469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.240703 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.240776 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.240912 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.241178 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"mutex_wait_us":41,"thread_start_us":1299,"threads_started":1}
I20260620 18:45:04.241930 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.227454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:36643 (local address 127.0.0.1:52894)
0620 18:45:04.232799 (+ 5345us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.232822 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.232941 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.233713 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.233720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.233774 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.234300 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.234322 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.236846 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.236851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.238491 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.238504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.241045 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.241051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.241214 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.241245 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.241336 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.241466 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5160,"mutex_wait_us":25,"spinlock_wait_cycles":25472,"thread_start_us":5003,"threads_started":1}
I20260620 18:45:04.242053 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.226428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36583 (local address 127.0.0.1:49444)
0620 18:45:04.227291 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.227340 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.227569 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.233959 (+ 6390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.233964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.234002 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.234503 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.234517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.236610 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.236615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.238898 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.238907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.240653 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.240661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.240803 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.240826 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.240887 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.241178 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":132,"spinlock_wait_cycles":13440,"thread_start_us":484,"threads_started":1}
I20260620 18:45:04.242698 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.228129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.15.102.195:36643)
0620 18:45:04.228916 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.228929 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.233060 (+ 4131us) server_negotiation.cc:407] Connection header received
0620 18:45:04.233182 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.233191 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.233465 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.233678 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.234538 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.236703 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.238657 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.240558 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.240756 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.240870 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.242218 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.242379 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":433,"threads_started":1}
I20260620 18:45:04.242724 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.226659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.15.102.193:36583)
0620 18:45:04.233410 (+ 6751us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.233418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.233442 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:45:04.233528 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.233534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.233735 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.234708 (+ 973us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.234911 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.236467 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.239103 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.240428 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.242228 (+ 1800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.242356 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.242411 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.242521 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6610,"thread_start_us":6470,"threads_started":1}
I20260620 18:45:04.245461 23629 tablet_service.cc:1467] Processing CreateTablet for tablet 5904e9cc782f449fa99632fda2aae7df (DEFAULT_TABLE table=test-table [id=dfa93dc747b149d1af12c02ca690a92b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.246170 23768 tablet_service.cc:1467] Processing CreateTablet for tablet 5904e9cc782f449fa99632fda2aae7df (DEFAULT_TABLE table=test-table [id=dfa93dc747b149d1af12c02ca690a92b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.246170 23489 tablet_service.cc:1467] Processing CreateTablet for tablet 5904e9cc782f449fa99632fda2aae7df (DEFAULT_TABLE table=test-table [id=dfa93dc747b149d1af12c02ca690a92b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.246869 23629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5904e9cc782f449fa99632fda2aae7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.247529 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5904e9cc782f449fa99632fda2aae7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.248308 23489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5904e9cc782f449fa99632fda2aae7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.258008 23862 tablet_bootstrap.cc:492] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: Bootstrap starting.
I20260620 18:45:04.261225 23862 tablet_bootstrap.cc:654] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.262121 23863 tablet_bootstrap.cc:492] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: Bootstrap starting.
I20260620 18:45:04.262660 23862 log.cc:824] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:04.263746 23864 tablet_bootstrap.cc:492] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Bootstrap starting.
I20260620 18:45:04.266152 23863 tablet_bootstrap.cc:654] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.266330 23864 tablet_bootstrap.cc:654] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.267343 23864 log.cc:824] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:04.267647 23863 log.cc:824] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:04.270263 23864 tablet_bootstrap.cc:492] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: No bootstrap required, opened a new log
I20260620 18:45:04.270526 23864 ts_tablet_manager.cc:1397] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 18:45:04.270961 23862 tablet_bootstrap.cc:492] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: No bootstrap required, opened a new log
I20260620 18:45:04.271214 23862 ts_tablet_manager.cc:1397] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260620 18:45:04.271309 23863 tablet_bootstrap.cc:492] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: No bootstrap required, opened a new log
I20260620 18:45:04.271548 23863 ts_tablet_manager.cc:1397] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:45:04.277325 23864 raft_consensus.cc:348] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.277691 23864 raft_consensus.cc:374] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.277787 23864 raft_consensus.cc:729] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bce56caa1f43e09c9ba1d857c1d24f, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.278391 23864 consensus_queue.cc:260] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.280195 23864 ts_tablet_manager.cc:1428] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260620 18:45:04.281709 23862 raft_consensus.cc:348] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.282124 23862 raft_consensus.cc:374] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.282217 23862 raft_consensus.cc:729] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606e725f5dbb4697bf8a1acae5b97617, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.282795 23862 consensus_queue.cc:260] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.282856 23863 raft_consensus.cc:348] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.283278 23863 raft_consensus.cc:374] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.283414 23863 raft_consensus.cc:729] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa02816cd3143e19c0ca41888ca44ed, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.283974 23699 heartbeater.cc:503] Master 127.15.102.254:37979 was elected leader, sending a full tablet report...
I20260620 18:45:04.284034 23863 consensus_queue.cc:260] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.286142 23862 ts_tablet_manager.cc:1428] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: Time spent starting tablet: real 0.015s user 0.008s sys 0.005s
I20260620 18:45:04.287052 23838 heartbeater.cc:503] Master 127.15.102.254:37979 was elected leader, sending a full tablet report...
I20260620 18:45:04.287745 23863 ts_tablet_manager.cc:1428] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
I20260620 18:45:04.293948 23868 raft_consensus.cc:482] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:04.294227 23868 raft_consensus.cc:504] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.296824 23868 leader_election.cc:288] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443), 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643)
I20260620 18:45:04.304482 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.297124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:42593 (local address 127.15.102.194:35443)
0620 18:45:04.297391 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.297401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.298468 (+ 1067us) server_negotiation.cc:407] Connection header received
0620 18:45:04.298651 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.298655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.298709 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.298834 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.299654 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.300681 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.302714 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.303573 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.303752 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.303836 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.304099 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.304216 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:45:04.304519 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.297529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:48213 (local address 127.15.102.195:36643)
0620 18:45:04.297765 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.297773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.298165 (+ 392us) server_negotiation.cc:407] Connection header received
0620 18:45:04.298434 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.298441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.298524 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.298624 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.299412 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.300477 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.302722 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.303634 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.303809 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.303887 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.304141 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.304259 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:45:04.304654 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.297126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:36643 (local address 127.15.102.193:48213)
0620 18:45:04.298028 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.298051 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.298266 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.298650 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.298655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.298683 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.299201 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.299215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.300620 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.300624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.302583 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.302595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.303866 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.303872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.304000 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.304021 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.304119 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.304241 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":97,"thread_start_us":878,"threads_started":1}
I20260620 18:45:04.304983 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.297173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35443 (local address 127.15.102.193:42593)
0620 18:45:04.298353 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.298367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.298511 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.298853 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.298858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.298888 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.299505 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.299518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.300807 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.300813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.302572 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.302584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.303776 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.303782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.303950 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.303973 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.304077 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.304202 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":48,"thread_start_us":465,"threads_started":1}
I20260620 18:45:04.305617 23793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5904e9cc782f449fa99632fda2aae7df" candidate_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aa02816cd3143e19c0ca41888ca44ed" is_pre_election: true
I20260620 18:45:04.306085 23793 raft_consensus.cc:2454] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9bce56caa1f43e09c9ba1d857c1d24f in term 0.
I20260620 18:45:04.306694 23441 leader_election.cc:302] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa02816cd3143e19c0ca41888ca44ed, b9bce56caa1f43e09c9ba1d857c1d24f; no voters:
I20260620 18:45:04.307106 23868 raft_consensus.cc:2781] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:04.307283 23868 raft_consensus.cc:482] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:04.307179 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5904e9cc782f449fa99632fda2aae7df" candidate_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606e725f5dbb4697bf8a1acae5b97617" is_pre_election: true
I20260620 18:45:04.307402 23868 raft_consensus.cc:3037] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.307721 23649 raft_consensus.cc:2454] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9bce56caa1f43e09c9ba1d857c1d24f in term 0.
I20260620 18:45:04.310215 23868 raft_consensus.cc:504] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.310739 23868 leader_election.cc:288] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [CANDIDATE]: Term 1 election: Requested vote from peers 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443), 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643)
I20260620 18:45:04.311123 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5904e9cc782f449fa99632fda2aae7df" candidate_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606e725f5dbb4697bf8a1acae5b97617"
I20260620 18:45:04.311164 23793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5904e9cc782f449fa99632fda2aae7df" candidate_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aa02816cd3143e19c0ca41888ca44ed"
I20260620 18:45:04.311340 23649 raft_consensus.cc:3037] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.311417 23793 raft_consensus.cc:3037] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.313529 23649 raft_consensus.cc:2454] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9bce56caa1f43e09c9ba1d857c1d24f in term 1.
I20260620 18:45:04.313609 23793 raft_consensus.cc:2454] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9bce56caa1f43e09c9ba1d857c1d24f in term 1.
I20260620 18:45:04.313927 23439 leader_election.cc:302] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606e725f5dbb4697bf8a1acae5b97617, b9bce56caa1f43e09c9ba1d857c1d24f; no voters:
I20260620 18:45:04.314196 23868 raft_consensus.cc:2781] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:04.314613 23868 raft_consensus.cc:686] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 LEADER]: Becoming Leader. State: Replica: b9bce56caa1f43e09c9ba1d857c1d24f, State: Running, Role: LEADER
I20260620 18:45:04.315049 23868 consensus_queue.cc:237] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.319660 23351 catalog_manager.cc:5495] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f reported cstate change: term changed from 0 to 1, leader changed from <none> to b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193). New cstate: current_term: 1 leader_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } health_report { overall_health: UNKNOWN } } }
W20260620 18:45:04.325822 23560 tablet.cc:2369] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:04.340418 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.335027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.15.102.193:36583)
0620 18:45:04.335230 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.335258 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.335404 (+ 146us) server_negotiation.cc:407] Connection header received
0620 18:45:04.335667 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.335671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.335752 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.335910 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:04.336260 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.336966 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.338302 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.338708 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.338868 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.339325 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.339849 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.340003 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.340081 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.340159 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:45:04.348373 23793 raft_consensus.cc:1261] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Refusing update from remote peer b9bce56caa1f43e09c9ba1d857c1d24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:04.348476 23649 raft_consensus.cc:1261] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Refusing update from remote peer b9bce56caa1f43e09c9ba1d857c1d24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:04.349408 23873 consensus_queue.cc:1035] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:04.349732 23868 consensus_queue.cc:1035] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:04.361799 23878 mvcc.cc:205] Tried to move back new op lower bound from 7298994603401625600 to 7298994603277832192. Current Snapshot: MvccSnapshot[applied={T|T < 7298994603401625600}]
I20260620 18:45:04.374109 23351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:04.374399 23351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:04.378052 23351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: Sending DeleteTablet for 3 replicas of tablet 5904e9cc782f449fa99632fda2aae7df
I20260620 18:45:04.379181 23489 tablet_service.cc:1514] Processing DeleteTablet for tablet 5904e9cc782f449fa99632fda2aae7df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:49444
I20260620 18:45:04.379274 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 5904e9cc782f449fa99632fda2aae7df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:52894
I20260620 18:45:04.380733 23351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:45:04.381600 23351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:04.383450 23629 tablet_service.cc:1514] Processing DeleteTablet for tablet 5904e9cc782f449fa99632fda2aae7df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:58446
I20260620 18:45:04.385100 23886 tablet_replica.cc:330] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: stopping tablet replica
I20260620 18:45:04.385528 23886 raft_consensus.cc:2229] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:04.386173 23886 raft_consensus.cc:2258] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.393100 23628 tablet_service.cc:1467] Processing CreateTablet for tablet 039e3cf8e91746449fabc9a91629d00f (DEFAULT_TABLE table=test-table [id=3a78a550c18b4a79af28f747458fd1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.393205 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 039e3cf8e91746449fabc9a91629d00f (DEFAULT_TABLE table=test-table [id=3a78a550c18b4a79af28f747458fd1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.393769 23628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039e3cf8e91746449fabc9a91629d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.393823 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039e3cf8e91746449fabc9a91629d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.394253 23886 ts_tablet_manager.cc:1905] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.396800 23888 tablet_replica.cc:330] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: stopping tablet replica
I20260620 18:45:04.397326 23888 raft_consensus.cc:2229] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:04.397912 23888 raft_consensus.cc:2258] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.398681 23887 tablet_replica.cc:330] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: stopping tablet replica
I20260620 18:45:04.399140 23887 raft_consensus.cc:2229] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:04.399657 23489 tablet_service.cc:1467] Processing CreateTablet for tablet 039e3cf8e91746449fabc9a91629d00f (DEFAULT_TABLE table=test-table [id=3a78a550c18b4a79af28f747458fd1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:04.399879 23887 raft_consensus.cc:2258] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.400094 23489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039e3cf8e91746449fabc9a91629d00f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.401167 23888 ts_tablet_manager.cc:1905] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.402056 23863 tablet_bootstrap.cc:492] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: Bootstrap starting.
I20260620 18:45:04.403719 23863 tablet_bootstrap.cc:654] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.404788 23886 ts_tablet_manager.cc:1918] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:04.404960 23886 log.cc:1196] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/5904e9cc782f449fa99632fda2aae7df
I20260620 18:45:04.405515 23886 ts_tablet_manager.cc:1939] T 5904e9cc782f449fa99632fda2aae7df P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting consensus metadata
I20260620 18:45:04.407152 23337 catalog_manager.cc:4841] TS b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583): tablet 5904e9cc782f449fa99632fda2aae7df (table test-table [id=dfa93dc747b149d1af12c02ca690a92b]) successfully deleted
I20260620 18:45:04.407717 23862 tablet_bootstrap.cc:492] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: Bootstrap starting.
I20260620 18:45:04.407747 23864 tablet_bootstrap.cc:492] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: Bootstrap starting.
I20260620 18:45:04.409332 23864 tablet_bootstrap.cc:654] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.409408 23862 tablet_bootstrap.cc:654] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.410529 23888 ts_tablet_manager.cc:1918] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:04.410753 23888 log.cc:1196] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/5904e9cc782f449fa99632fda2aae7df
I20260620 18:45:04.411413 23888 ts_tablet_manager.cc:1939] T 5904e9cc782f449fa99632fda2aae7df P 606e725f5dbb4697bf8a1acae5b97617: Deleting consensus metadata
I20260620 18:45:04.413367 23336 catalog_manager.cc:4841] TS 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443): tablet 5904e9cc782f449fa99632fda2aae7df (table test-table [id=dfa93dc747b149d1af12c02ca690a92b]) successfully deleted
I20260620 18:45:04.414947 23862 tablet_bootstrap.cc:492] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: No bootstrap required, opened a new log
I20260620 18:45:04.415076 23862 ts_tablet_manager.cc:1397] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260620 18:45:04.415947 23862 raft_consensus.cc:348] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.416174 23862 raft_consensus.cc:374] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.416282 23862 raft_consensus.cc:729] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606e725f5dbb4697bf8a1acae5b97617, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.416478 23864 tablet_bootstrap.cc:492] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: No bootstrap required, opened a new log
I20260620 18:45:04.416620 23864 ts_tablet_manager.cc:1397] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260620 18:45:04.416565 23862 consensus_queue.cc:260] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.417222 23887 ts_tablet_manager.cc:1905] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.417402 23864 raft_consensus.cc:348] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.417631 23864 raft_consensus.cc:374] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.417729 23864 raft_consensus.cc:729] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bce56caa1f43e09c9ba1d857c1d24f, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.417982 23864 consensus_queue.cc:260] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.419063 23864 ts_tablet_manager.cc:1428] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:45:04.423260 23862 ts_tablet_manager.cc:1428] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260620 18:45:04.423769 23863 tablet_bootstrap.cc:492] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: No bootstrap required, opened a new log
I20260620 18:45:04.423950 23863 ts_tablet_manager.cc:1397] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.005s
I20260620 18:45:04.424800 23863 raft_consensus.cc:348] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.425035 23863 raft_consensus.cc:374] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.425130 23863 raft_consensus.cc:729] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa02816cd3143e19c0ca41888ca44ed, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.425462 23863 consensus_queue.cc:260] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.425915 23887 ts_tablet_manager.cc:1918] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:04.426082 23887 log.cc:1196] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/5904e9cc782f449fa99632fda2aae7df
I20260620 18:45:04.426172 23863 ts_tablet_manager.cc:1428] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 18:45:04.426646 23887 ts_tablet_manager.cc:1939] T 5904e9cc782f449fa99632fda2aae7df P 0aa02816cd3143e19c0ca41888ca44ed: Deleting consensus metadata
I20260620 18:45:04.428364 23338 catalog_manager.cc:4841] TS 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643): tablet 5904e9cc782f449fa99632fda2aae7df (table test-table [id=dfa93dc747b149d1af12c02ca690a92b]) successfully deleted
I20260620 18:45:04.440533 23869 raft_consensus.cc:482] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:04.440773 23869 raft_consensus.cc:504] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.442806 23869 leader_election.cc:288] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583), 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643)
I20260620 18:45:04.451126 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.443566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:51803 (local address 127.15.102.193:36583)
0620 18:45:04.443737 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.443743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.444711 (+ 968us) server_negotiation.cc:407] Connection header received
0620 18:45:04.444890 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.444898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.444970 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.445016 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.445771 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.446947 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.449264 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.450053 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.450237 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.450325 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.450659 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.450803 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:45:04.451509 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.443083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:36643 (local address 127.15.102.194:33747)
0620 18:45:04.444190 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.444209 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.444409 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.444806 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.444811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.444839 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.445216 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.445231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.446608 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.446613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.448912 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.448927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.450596 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.450602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.450764 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.450793 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.450863 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.450996 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":38,"thread_start_us":1205,"threads_started":1}
I20260620 18:45:04.451854 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.443154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36583 (local address 127.15.102.194:51803)
0620 18:45:04.444602 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.444619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.444738 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.445157 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.445160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.445181 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.445587 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.445601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.447087 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.447093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.449122 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.449131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.450253 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.450260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.450453 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.450494 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.451383 (+ 889us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.451528 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"thread_start_us":753,"threads_started":1}
I20260620 18:45:04.452740 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039e3cf8e91746449fabc9a91629d00f" candidate_uuid: "606e725f5dbb4697bf8a1acae5b97617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" is_pre_election: true
I20260620 18:45:04.453215 23509 raft_consensus.cc:2454] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606e725f5dbb4697bf8a1acae5b97617 in term 0.
I20260620 18:45:04.453344 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.443394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:33747 (local address 127.15.102.195:36643)
0620 18:45:04.443574 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.443580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.444293 (+ 713us) server_negotiation.cc:407] Connection header received
0620 18:45:04.444572 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.444578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.444663 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.444746 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.445471 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.446473 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.449086 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.450327 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.452502 (+ 2175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.452613 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.452694 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.452919 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:45:04.453920 23580 leader_election.cc:302] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606e725f5dbb4697bf8a1acae5b97617, b9bce56caa1f43e09c9ba1d857c1d24f; no voters:
I20260620 18:45:04.454118 23793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039e3cf8e91746449fabc9a91629d00f" candidate_uuid: "606e725f5dbb4697bf8a1acae5b97617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aa02816cd3143e19c0ca41888ca44ed" is_pre_election: true
I20260620 18:45:04.454478 23793 raft_consensus.cc:2454] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606e725f5dbb4697bf8a1acae5b97617 in term 0.
I20260620 18:45:04.455732 23869 raft_consensus.cc:2781] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:04.455864 23869 raft_consensus.cc:482] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:04.455962 23869 raft_consensus.cc:3037] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.459468 23869 raft_consensus.cc:504] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.460573 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039e3cf8e91746449fabc9a91629d00f" candidate_uuid: "606e725f5dbb4697bf8a1acae5b97617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f"
I20260620 18:45:04.460656 23793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039e3cf8e91746449fabc9a91629d00f" candidate_uuid: "606e725f5dbb4697bf8a1acae5b97617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aa02816cd3143e19c0ca41888ca44ed"
I20260620 18:45:04.460764 23509 raft_consensus.cc:3037] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.460865 23793 raft_consensus.cc:3037] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.463773 23793 raft_consensus.cc:2454] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606e725f5dbb4697bf8a1acae5b97617 in term 1.
I20260620 18:45:04.464097 23509 raft_consensus.cc:2454] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606e725f5dbb4697bf8a1acae5b97617 in term 1.
I20260620 18:45:04.464218 23581 leader_election.cc:302] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa02816cd3143e19c0ca41888ca44ed, 606e725f5dbb4697bf8a1acae5b97617; no voters:
I20260620 18:45:04.464826 23869 leader_election.cc:288] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [CANDIDATE]: Term 1 election: Requested vote from peers b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583), 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643)
I20260620 18:45:04.465034 23869 raft_consensus.cc:2781] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:04.465161 23869 raft_consensus.cc:686] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 LEADER]: Becoming Leader. State: Replica: 606e725f5dbb4697bf8a1acae5b97617, State: Running, Role: LEADER
I20260620 18:45:04.465538 23869 consensus_queue.cc:237] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.471484 23351 catalog_manager.cc:5495] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194). New cstate: current_term: 1 leader_uuid: "606e725f5dbb4697bf8a1acae5b97617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } health_report { overall_health: UNKNOWN } } }
Jun 20 18:45:04 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981102, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.254@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:45:04.537230 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.522010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37979 (local address 127.0.0.1:34792)
0620 18:45:04.522714 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.522732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.522862 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.523244 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.523248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.524092 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:04.524389 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.524402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.525820 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.525823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.526801 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.526812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.527843 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.527847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.527970 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.528689 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:04.528705 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:04.530872 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:04.534762 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:04.534774 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:04.534780 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:04.535219 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:04.535667 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:04.535671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:04.535673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:04.535828 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:04.536395 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:04.536401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:04.536552 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.536684 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.536914 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":375,"threads_started":1}
I20260620 18:45:04.537508 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.521994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.15.102.254:37979)
0620 18:45:04.522225 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.522232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.522833 (+ 601us) server_negotiation.cc:407] Connection header received
0620 18:45:04.522982 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.522990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.523070 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.523190 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:04.524549 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.525693 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.526937 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.527652 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.527774 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.531068 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:04.531075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:04.531080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:04.531118 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:04.534587 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.535399 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.535407 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.535411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.535553 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:04.535934 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:04.535939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:04.535942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:04.536271 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:04.536422 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.536799 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.537070 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":76}
I20260620 18:45:04.542632 23351 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:34792:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 18:45:04.543782 23351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:04.550805 23489 tablet_service.cc:1467] Processing CreateTablet for tablet 230cfa0ce2ab441aa83bf7fb193fa39d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cce76898216d4840a52afc9d02ee901d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:45:04.550982 23628 tablet_service.cc:1467] Processing CreateTablet for tablet 230cfa0ce2ab441aa83bf7fb193fa39d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cce76898216d4840a52afc9d02ee901d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:45:04.551478 23489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230cfa0ce2ab441aa83bf7fb193fa39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.551589 23628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230cfa0ce2ab441aa83bf7fb193fa39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.556025 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 230cfa0ce2ab441aa83bf7fb193fa39d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cce76898216d4840a52afc9d02ee901d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 18:45:04.556556 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230cfa0ce2ab441aa83bf7fb193fa39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:04.557608 23862 tablet_bootstrap.cc:492] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: Bootstrap starting.
I20260620 18:45:04.558895 23862 tablet_bootstrap.cc:654] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.561507 23862 tablet_bootstrap.cc:492] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: No bootstrap required, opened a new log
I20260620 18:45:04.561697 23862 ts_tablet_manager.cc:1397] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 18:45:04.562403 23862 raft_consensus.cc:348] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.562619 23862 raft_consensus.cc:374] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.562722 23862 raft_consensus.cc:729] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606e725f5dbb4697bf8a1acae5b97617, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.562990 23862 consensus_queue.cc:260] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.563597 23862 ts_tablet_manager.cc:1428] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 18:45:04.563476 23894 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.563905 23894 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:45:04.567845 23864 tablet_bootstrap.cc:492] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: Bootstrap starting.
I20260620 18:45:04.568596 23863 tablet_bootstrap.cc:492] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: Bootstrap starting.
I20260620 18:45:04.569371 23864 tablet_bootstrap.cc:654] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.570286 23863 tablet_bootstrap.cc:654] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:04.571820 23864 tablet_bootstrap.cc:492] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: No bootstrap required, opened a new log
I20260620 18:45:04.572001 23864 ts_tablet_manager.cc:1397] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 18:45:04.572319 23863 tablet_bootstrap.cc:492] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: No bootstrap required, opened a new log
I20260620 18:45:04.572474 23863 ts_tablet_manager.cc:1397] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 18:45:04.572713 23864 raft_consensus.cc:348] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.572943 23864 raft_consensus.cc:374] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.573045 23864 raft_consensus.cc:729] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bce56caa1f43e09c9ba1d857c1d24f, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.573220 23863 raft_consensus.cc:348] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.573297 23864 consensus_queue.cc:260] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.573454 23863 raft_consensus.cc:374] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:04.573928 23864 ts_tablet_manager.cc:1428] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 18:45:04.573719 23873 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.574105 23873 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:45:04.574410 23863 raft_consensus.cc:729] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa02816cd3143e19c0ca41888ca44ed, State: Initialized, Role: FOLLOWER
I20260620 18:45:04.574707 23863 consensus_queue.cc:260] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.575158 23870 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.575533 23870 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:45:04.577327 23863 ts_tablet_manager.cc:1428] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20260620 18:45:04.624714 23700 tablet.cc:2369] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:45:04.635528 23839 tablet.cc:2369] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:04.688316 23870 raft_consensus.cc:482] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:04.688572 23870 raft_consensus.cc:504] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.690529 23870 leader_election.cc:288] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583), 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443)
I20260620 18:45:04.699064 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.691387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:55429 (local address 127.15.102.194:35443)
0620 18:45:04.691567 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.691573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.692233 (+ 660us) server_negotiation.cc:407] Connection header received
0620 18:45:04.692808 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.692816 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.692897 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.692972 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.693642 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.694737 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.697094 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.698048 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.698239 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.698339 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.698650 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.698778 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:45:04.699173 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.690740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35443 (local address 127.15.102.195:55429)
0620 18:45:04.692060 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.692083 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.692644 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.693091 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.693095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.693117 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.693473 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.693488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.694880 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.694885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.696885 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.696894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.698224 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.698230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.698479 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.698521 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.698626 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.698818 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":2,"thread_start_us":1301,"threads_started":1}
I20260620 18:45:04.699546 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.690990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:58475 (local address 127.15.102.193:36583)
0620 18:45:04.691202 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.691210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.692421 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 18:45:04.692670 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.692677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.692773 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.692880 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:04.693411 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.694261 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.696676 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.698125 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.698294 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.698385 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.699041 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.699191 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:45:04.699800 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230cfa0ce2ab441aa83bf7fb193fa39d" candidate_uuid: "0aa02816cd3143e19c0ca41888ca44ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606e725f5dbb4697bf8a1acae5b97617" is_pre_election: true
I20260620 18:45:04.700050 23649 raft_consensus.cc:2454] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa02816cd3143e19c0ca41888ca44ed in term 0.
I20260620 18:45:04.700444 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.690740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36583 (local address 127.15.102.195:58475)
0620 18:45:04.692319 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.692332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.692530 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.692920 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.692926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.692953 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:04.693226 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.693238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.694401 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.694409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.696485 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.696499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.698571 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.698578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.698817 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.698862 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.699914 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.700081 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1473,"mutex_wait_us":139,"thread_start_us":838,"threads_started":1}
I20260620 18:45:04.700546 23719 leader_election.cc:302] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa02816cd3143e19c0ca41888ca44ed, 606e725f5dbb4697bf8a1acae5b97617; no voters:
I20260620 18:45:04.700979 23870 raft_consensus.cc:2781] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:04.701115 23870 raft_consensus.cc:482] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:04.701225 23870 raft_consensus.cc:3037] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.701215 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230cfa0ce2ab441aa83bf7fb193fa39d" candidate_uuid: "0aa02816cd3143e19c0ca41888ca44ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" is_pre_election: true
I20260620 18:45:04.701474 23509 raft_consensus.cc:2454] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa02816cd3143e19c0ca41888ca44ed in term 0.
I20260620 18:45:04.703765 23870 raft_consensus.cc:504] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.704334 23870 leader_election.cc:288] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [CANDIDATE]: Term 1 election: Requested vote from peers b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583), 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443)
I20260620 18:45:04.704768 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230cfa0ce2ab441aa83bf7fb193fa39d" candidate_uuid: "0aa02816cd3143e19c0ca41888ca44ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f"
I20260620 18:45:04.704981 23509 raft_consensus.cc:3037] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.707582 23509 raft_consensus.cc:2454] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa02816cd3143e19c0ca41888ca44ed in term 1.
I20260620 18:45:04.708027 23720 leader_election.cc:302] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa02816cd3143e19c0ca41888ca44ed, b9bce56caa1f43e09c9ba1d857c1d24f; no voters:
I20260620 18:45:04.708112 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230cfa0ce2ab441aa83bf7fb193fa39d" candidate_uuid: "0aa02816cd3143e19c0ca41888ca44ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606e725f5dbb4697bf8a1acae5b97617"
I20260620 18:45:04.708338 23870 raft_consensus.cc:2781] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:04.708374 23649 raft_consensus.cc:3037] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:04.712242 23649 raft_consensus.cc:2454] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa02816cd3143e19c0ca41888ca44ed in term 1.
I20260620 18:45:04.712739 23870 raft_consensus.cc:686] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [term 1 LEADER]: Becoming Leader. State: Replica: 0aa02816cd3143e19c0ca41888ca44ed, State: Running, Role: LEADER
I20260620 18:45:04.713363 23870 consensus_queue.cc:237] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:04.715655 23907 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: TxnStatusTablet state changed. Reason: New leader 0aa02816cd3143e19c0ca41888ca44ed. Latest consensus state: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.716085 23907 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:45:04.716785 23909 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 18:45:04.717028 23909 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 18:45:04.717248 23351 catalog_manager.cc:5495] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195). New cstate: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } health_report { overall_health: HEALTHY } } }
I20260620 18:45:04.756510 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.750745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.15.102.195:36643)
0620 18:45:04.751403 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.751408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.751423 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:45:04.751497 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.751501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.751569 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.751721 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:04.752112 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.752958 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.754205 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.754698 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.754851 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.755321 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.755839 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.755993 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.756078 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.756171 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542}
I20260620 18:45:04.756497 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.750688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:36643 (local address 127.0.0.1:52908)
0620 18:45:04.750989 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.751006 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.751108 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.751750 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.751755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.751766 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:45:04.751970 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.751979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.753085 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.753092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.754073 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.754084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.754848 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.754854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.755043 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.755095 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.755959 (+ 864us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.755979 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.756060 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.756216 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 18:45:04.760497 23649 raft_consensus.cc:1261] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Refusing update from remote peer 0aa02816cd3143e19c0ca41888ca44ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:04.760749 23509 raft_consensus.cc:1261] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Refusing update from remote peer 0aa02816cd3143e19c0ca41888ca44ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:04.761531 23870 consensus_queue.cc:1035] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:04.762202 23870 consensus_queue.cc:1035] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260620 18:45:04.772785 23894 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: TxnStatusTablet state changed. Reason: New leader 0aa02816cd3143e19c0ca41888ca44ed. Latest consensus state: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.773149 23894 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:45:04.780212 23873 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: TxnStatusTablet state changed. Reason: New leader 0aa02816cd3143e19c0ca41888ca44ed. Latest consensus state: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.780539 23873 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:45:04.783592 23870 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.783936 23870 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:45:04.785954 23894 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.786186 23894 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P 606e725f5dbb4697bf8a1acae5b97617: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:45:04.791471 23873 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.791741 23873 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P b9bce56caa1f43e09c9ba1d857c1d24f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 18:45:04.792043 23914 tablet_replica.cc:439] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } } }
I20260620 18:45:04.792398 23914 tablet_replica.cc:442] T 230cfa0ce2ab441aa83bf7fb193fa39d P 0aa02816cd3143e19c0ca41888ca44ed: This TxnStatusTablet replica's current role is: LEADER
I20260620 18:45:04.796186 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.786352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.15.102.194:35443)
0620 18:45:04.786523 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.786528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.786544 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:45:04.786633 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.786638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.786758 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.786914 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:04.788097 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.789108 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.790952 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.793644 (+ 2692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.793768 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.794783 (+ 1015us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.795462 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.795560 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.795765 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.795851 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:45:04.812160 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.806560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:44389 (local address 127.15.102.195:36643)
0620 18:45:04.806767 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.806773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.806881 (+ 108us) server_negotiation.cc:407] Connection header received
0620 18:45:04.807152 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.807157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.807288 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.807369 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:04.807854 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.808715 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.810293 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.810793 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.810945 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.811400 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.811502 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.811598 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.811740 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.811819 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:45:04.812214 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.806259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:36643 (local address 127.15.102.194:44389)
0620 18:45:04.806785 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.806805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.806997 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.807409 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.807414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.807426 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:45:04.807701 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.807709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.808858 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.808864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.810133 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.810145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.810967 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.810973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.811120 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.811176 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.811598 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.811611 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.811712 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.811839 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":527,"threads_started":1}
I20260620 18:45:04.823082 23793 raft_consensus.cc:1261] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Refusing update from remote peer 606e725f5dbb4697bf8a1acae5b97617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:04.823082 23509 raft_consensus.cc:1261] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Refusing update from remote peer 606e725f5dbb4697bf8a1acae5b97617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:04.823832 23869 consensus_queue.cc:1035] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:04.824198 23894 consensus_queue.cc:1035] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:04.860323 23877 mvcc.cc:205] Tried to move back new op lower bound from 7298994605347221504 to 7298994603895021568. Current Snapshot: MvccSnapshot[applied={T|T < 7298994605347221504}]
I20260620 18:45:04.879034 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.871795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:48181 (local address 127.15.102.194:35443)
0620 18:45:04.871999 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:04.872006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:04.872027 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:45:04.872102 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:04.872108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:04.872203 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:04.872325 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:04.873057 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.873915 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.876786 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.877429 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.877518 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.878155 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.878266 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.878384 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:04.878560 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 18:45:04.878657 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:45:04.879361 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:04.870475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35443 (local address 127.15.102.195:48181)
0620 18:45:04.871314 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:04.871338 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:04.871489 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:04.872345 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:04.872351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:04.872365 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:45:04.872811 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.872833 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.874082 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.874086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.876586 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:04.876604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:04.877701 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:04.877706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:04.877849 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:04.877910 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:45:04.878390 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:45:04.878405 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 18:45:04.878537 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 18:45:04.878687 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":2373,"threads_started":1}
I20260620 18:45:04.934998 23352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:04.935340 23352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:04.938841 23352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: Sending DeleteTablet for 3 replicas of tablet 039e3cf8e91746449fabc9a91629d00f
I20260620 18:45:04.939839 23489 tablet_service.cc:1514] Processing DeleteTablet for tablet 039e3cf8e91746449fabc9a91629d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:49444
I20260620 18:45:04.940276 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 039e3cf8e91746449fabc9a91629d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:52894
I20260620 18:45:04.940990 23925 tablet_replica.cc:330] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: stopping tablet replica
I20260620 18:45:04.941066 23628 tablet_service.cc:1514] Processing DeleteTablet for tablet 039e3cf8e91746449fabc9a91629d00f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:04 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:58446
I20260620 18:45:04.941289 23925 raft_consensus.cc:2229] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:04.941565 23925 raft_consensus.cc:2258] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.951603 23926 tablet_replica.cc:330] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: stopping tablet replica
I20260620 18:45:04.952446 23924 tablet_replica.cc:330] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: stopping tablet replica
I20260620 18:45:04.952759 23924 raft_consensus.cc:2229] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:04.953037 23924 raft_consensus.cc:2258] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.952698 23926 raft_consensus.cc:2229] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:04.954202 23926 raft_consensus.cc:2258] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:04.954262 23924 ts_tablet_manager.cc:1905] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.955561 23926 ts_tablet_manager.cc:1905] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.962616 23924 ts_tablet_manager.cc:1918] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:45:04.963176 23924 log.cc:1196] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/039e3cf8e91746449fabc9a91629d00f
I20260620 18:45:04.963747 23924 ts_tablet_manager.cc:1939] T 039e3cf8e91746449fabc9a91629d00f P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting consensus metadata
I20260620 18:45:04.963946 23925 ts_tablet_manager.cc:1905] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:04.965095 23337 catalog_manager.cc:4841] TS b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583): tablet 039e3cf8e91746449fabc9a91629d00f (table test-table [id=3a78a550c18b4a79af28f747458fd1db]) successfully deleted
I20260620 18:45:04.969584 23926 ts_tablet_manager.cc:1918] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:45:04.969774 23926 log.cc:1196] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/039e3cf8e91746449fabc9a91629d00f
I20260620 18:45:04.970309 23926 ts_tablet_manager.cc:1939] T 039e3cf8e91746449fabc9a91629d00f P 606e725f5dbb4697bf8a1acae5b97617: Deleting consensus metadata
I20260620 18:45:04.971778 23336 catalog_manager.cc:4841] TS 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443): tablet 039e3cf8e91746449fabc9a91629d00f (table test-table [id=3a78a550c18b4a79af28f747458fd1db]) successfully deleted
I20260620 18:45:04.974984 23925 ts_tablet_manager.cc:1918] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:45:04.975196 23925 log.cc:1196] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/039e3cf8e91746449fabc9a91629d00f
I20260620 18:45:04.975855 23925 ts_tablet_manager.cc:1939] T 039e3cf8e91746449fabc9a91629d00f P 0aa02816cd3143e19c0ca41888ca44ed: Deleting consensus metadata
I20260620 18:45:04.977335 23338 catalog_manager.cc:4841] TS 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643): tablet 039e3cf8e91746449fabc9a91629d00f (table test-table [id=3a78a550c18b4a79af28f747458fd1db]) successfully deleted
Jun 20 18:45:05 dist-test-slave-zfp8 krb5kdc[23307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 18:45:05 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:05 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:45:05.030092 23587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:05.013995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.15.102.254:37979)
0620 18:45:05.014225 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.014233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.014779 (+ 546us) server_negotiation.cc:407] Connection header received
0620 18:45:05.015054 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.015059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.015134 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.015297 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:05.016772 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.017914 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.019767 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.020252 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.020421 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.029791 (+ 9370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34800: BlockingRecv error: recv got EOF from 127.0.0.1:34800 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:45:05 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM
I20260620 18:45:05.051460 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.034220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.15.102.254:37979)
0620 18:45:05.034430 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.034439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.035437 (+ 998us) server_negotiation.cc:407] Connection header received
0620 18:45:05.035698 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.035702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.035775 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.035904 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:05.037428 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.038272 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.039811 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.040316 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.040489 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.044781 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:05.044786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:05.044790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:05.044845 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:05.048139 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.048951 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.048958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.048961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.049113 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.049579 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.049591 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.049594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.050008 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:05.050324 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:05.050691 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 18:45:05.050923 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:45:05.055758 23352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:45:05.056599 23352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:05.064898 23489 tablet_service.cc:1467] Processing CreateTablet for tablet 835120b6115747db8bd40097a13e6915 (DEFAULT_TABLE table=test-table [id=ad8c4447dc864319b3f44c47c3d6e915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:05.065166 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 835120b6115747db8bd40097a13e6915 (DEFAULT_TABLE table=test-table [id=ad8c4447dc864319b3f44c47c3d6e915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:05.065423 23489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835120b6115747db8bd40097a13e6915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:05.065706 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835120b6115747db8bd40097a13e6915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:05.071985 23628 tablet_service.cc:1467] Processing CreateTablet for tablet 835120b6115747db8bd40097a13e6915 (DEFAULT_TABLE table=test-table [id=ad8c4447dc864319b3f44c47c3d6e915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:05.072561 23628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835120b6115747db8bd40097a13e6915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:05.077569 23939 tablet_bootstrap.cc:492] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: Bootstrap starting.
I20260620 18:45:05.080600 23939 tablet_bootstrap.cc:654] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:05.086728 23863 tablet_bootstrap.cc:492] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: Bootstrap starting.
I20260620 18:45:05.088920 23863 tablet_bootstrap.cc:654] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:05.095741 23941 tablet_bootstrap.cc:492] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: Bootstrap starting.
I20260620 18:45:05.097326 23941 tablet_bootstrap.cc:654] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:05.100463 23939 tablet_bootstrap.cc:492] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: No bootstrap required, opened a new log
I20260620 18:45:05.101169 23939 ts_tablet_manager.cc:1397] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.012s
I20260620 18:45:05.103502 23939 raft_consensus.cc:348] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.104007 23939 raft_consensus.cc:374] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:05.104192 23939 raft_consensus.cc:729] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bce56caa1f43e09c9ba1d857c1d24f, State: Initialized, Role: FOLLOWER
I20260620 18:45:05.104568 23863 tablet_bootstrap.cc:492] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: No bootstrap required, opened a new log
I20260620 18:45:05.104743 23863 ts_tablet_manager.cc:1397] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.007s
I20260620 18:45:05.104588 23939 consensus_queue.cc:260] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.105509 23863 raft_consensus.cc:348] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.105810 23863 raft_consensus.cc:374] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:05.105911 23863 raft_consensus.cc:729] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa02816cd3143e19c0ca41888ca44ed, State: Initialized, Role: FOLLOWER
I20260620 18:45:05.106156 23863 consensus_queue.cc:260] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.106720 23863 ts_tablet_manager.cc:1428] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 18:45:05.109490 23941 tablet_bootstrap.cc:492] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: No bootstrap required, opened a new log
I20260620 18:45:05.109635 23941 ts_tablet_manager.cc:1397] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20260620 18:45:05.110301 23941 raft_consensus.cc:348] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.110524 23941 raft_consensus.cc:374] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:05.110625 23941 raft_consensus.cc:729] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606e725f5dbb4697bf8a1acae5b97617, State: Initialized, Role: FOLLOWER
I20260620 18:45:05.110873 23941 consensus_queue.cc:260] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.112680 23939 ts_tablet_manager.cc:1428] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20260620 18:45:05.113510 23941 ts_tablet_manager.cc:1428] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 18:45:05.164253 23869 raft_consensus.cc:482] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:05.164443 23869 raft_consensus.cc:504] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.165215 23869 leader_election.cc:288] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583), 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643)
I20260620 18:45:05.166056 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835120b6115747db8bd40097a13e6915" candidate_uuid: "606e725f5dbb4697bf8a1acae5b97617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" is_pre_election: true
I20260620 18:45:05.166299 23509 raft_consensus.cc:2454] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606e725f5dbb4697bf8a1acae5b97617 in term 0.
I20260620 18:45:05.166915 23793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835120b6115747db8bd40097a13e6915" candidate_uuid: "606e725f5dbb4697bf8a1acae5b97617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aa02816cd3143e19c0ca41888ca44ed" is_pre_election: true
I20260620 18:45:05.167147 23793 raft_consensus.cc:2454] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 606e725f5dbb4697bf8a1acae5b97617 in term 0.
I20260620 18:45:05.167627 23580 leader_election.cc:302] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606e725f5dbb4697bf8a1acae5b97617, b9bce56caa1f43e09c9ba1d857c1d24f; no voters:
I20260620 18:45:05.168243 23869 raft_consensus.cc:2781] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:05.168368 23869 raft_consensus.cc:482] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:05.168462 23869 raft_consensus.cc:3037] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:05.172307 23869 raft_consensus.cc:504] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.173307 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835120b6115747db8bd40097a13e6915" candidate_uuid: "606e725f5dbb4697bf8a1acae5b97617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f"
I20260620 18:45:05.173521 23509 raft_consensus.cc:3037] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:05.176524 23509 raft_consensus.cc:2454] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606e725f5dbb4697bf8a1acae5b97617 in term 1.
I20260620 18:45:05.176985 23580 leader_election.cc:302] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 606e725f5dbb4697bf8a1acae5b97617, b9bce56caa1f43e09c9ba1d857c1d24f; no voters:
I20260620 18:45:05.177280 23894 raft_consensus.cc:2781] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:05.180205 23793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835120b6115747db8bd40097a13e6915" candidate_uuid: "606e725f5dbb4697bf8a1acae5b97617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0aa02816cd3143e19c0ca41888ca44ed"
I20260620 18:45:05.180447 23793 raft_consensus.cc:3037] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:05.181731 23869 leader_election.cc:288] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [CANDIDATE]: Term 1 election: Requested vote from peers b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583), 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643)
I20260620 18:45:05.183992 23793 raft_consensus.cc:2454] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 606e725f5dbb4697bf8a1acae5b97617 in term 1.
I20260620 18:45:05.186532 23894 raft_consensus.cc:686] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 1 LEADER]: Becoming Leader. State: Replica: 606e725f5dbb4697bf8a1acae5b97617, State: Running, Role: LEADER
I20260620 18:45:05.186832 23894 consensus_queue.cc:237] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.189868 23352 catalog_manager.cc:5495] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 reported cstate change: term changed from 0 to 1, leader changed from <none> to 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194). New cstate: current_term: 1 leader_uuid: "606e725f5dbb4697bf8a1acae5b97617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } health_report { overall_health: UNKNOWN } } }
I20260620 18:45:05.216559 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.209447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.15.102.194:35443)
0620 18:45:05.209643 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.209649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.209669 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:45:05.209870 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.209876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.209964 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.210063 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:05.210552 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.211737 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.213949 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.214641 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.214769 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.215491 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:05.215626 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:05.215714 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:05.216202 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 18:45:05.216282 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:45:05.219736 23793 raft_consensus.cc:1261] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Refusing update from remote peer 606e725f5dbb4697bf8a1acae5b97617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:05.220072 23509 raft_consensus.cc:1261] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Refusing update from remote peer 606e725f5dbb4697bf8a1acae5b97617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:05.220381 23944 consensus_queue.cc:1035] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:05.221101 23944 consensus_queue.cc:1035] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:05.240231 23350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:05.240636 23350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:05.247064 23350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: Sending DeleteTablet for 3 replicas of tablet 835120b6115747db8bd40097a13e6915
I20260620 18:45:05.248319 23628 tablet_service.cc:1514] Processing DeleteTablet for tablet 835120b6115747db8bd40097a13e6915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:05 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:58446
I20260620 18:45:05.248437 23489 tablet_service.cc:1514] Processing DeleteTablet for tablet 835120b6115747db8bd40097a13e6915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:05 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:49444
I20260620 18:45:05.248783 23488 tablet_service.cc:1514] Processing DeleteTablet for tablet 835120b6115747db8bd40097a13e6915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:05 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:49444
I20260620 18:45:05.249150 23926 tablet_replica.cc:330] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: stopping tablet replica
I20260620 18:45:05.249429 23926 raft_consensus.cc:2229] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:05.249856 23926 raft_consensus.cc:2258] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:05.250197 23924 tablet_replica.cc:330] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: stopping tablet replica
I20260620 18:45:05.250434 23924 raft_consensus.cc:2229] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:05.250717 23924 raft_consensus.cc:2258] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:05.252279 23924 ts_tablet_manager.cc:1905] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:05.255985 23767 tablet_service.cc:1514] Processing DeleteTablet for tablet 835120b6115747db8bd40097a13e6915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:05 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:52894
I20260620 18:45:05.256348 23925 tablet_replica.cc:330] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: stopping tablet replica
I20260620 18:45:05.256590 23925 raft_consensus.cc:2229] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:05.256892 23925 raft_consensus.cc:2258] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:05.257148 23926 ts_tablet_manager.cc:1905] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:05.261377 23925 ts_tablet_manager.cc:1905] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:05.264483 23926 ts_tablet_manager.cc:1918] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:05.264948 23352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 18:45:05.265542 23926 log.cc:1196] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/835120b6115747db8bd40097a13e6915
W20260620 18:45:05.266080 23352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:05.266686 23926 ts_tablet_manager.cc:1939] T 835120b6115747db8bd40097a13e6915 P 606e725f5dbb4697bf8a1acae5b97617: Deleting consensus metadata
I20260620 18:45:05.269223 23336 catalog_manager.cc:4841] TS 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443): tablet 835120b6115747db8bd40097a13e6915 (table test-table [id=ad8c4447dc864319b3f44c47c3d6e915]) successfully deleted
I20260620 18:45:05.273175 23924 ts_tablet_manager.cc:1918] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:05.273422 23924 log.cc:1196] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/835120b6115747db8bd40097a13e6915
I20260620 18:45:05.273892 23924 ts_tablet_manager.cc:1939] T 835120b6115747db8bd40097a13e6915 P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting consensus metadata
I20260620 18:45:05.276644 23628 tablet_service.cc:1467] Processing CreateTablet for tablet 3636f6bdeab648508116ec4ebfe2c85f (DEFAULT_TABLE table=test-table [id=2e14f73d1aa149cc9bad7d7d3f7496e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:05.277206 23628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3636f6bdeab648508116ec4ebfe2c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:05.277652 23925 ts_tablet_manager.cc:1918] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:05.277882 23925 log.cc:1196] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/835120b6115747db8bd40097a13e6915
I20260620 18:45:05.278421 23925 ts_tablet_manager.cc:1939] T 835120b6115747db8bd40097a13e6915 P 0aa02816cd3143e19c0ca41888ca44ed: Deleting consensus metadata
I20260620 18:45:05.279644 23338 catalog_manager.cc:4841] TS 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643): tablet 835120b6115747db8bd40097a13e6915 (table test-table [id=ad8c4447dc864319b3f44c47c3d6e915]) successfully deleted
I20260620 18:45:05.279793 23767 tablet_service.cc:1467] Processing CreateTablet for tablet 3636f6bdeab648508116ec4ebfe2c85f (DEFAULT_TABLE table=test-table [id=2e14f73d1aa149cc9bad7d7d3f7496e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:05.280298 23767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3636f6bdeab648508116ec4ebfe2c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:05.280514 23337 catalog_manager.cc:4841] TS b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583): tablet 835120b6115747db8bd40097a13e6915 (table test-table [id=ad8c4447dc864319b3f44c47c3d6e915]) successfully deleted
W20260620 18:45:05.280910 23337 catalog_manager.cc:4808] TS b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583): delete failed for tablet 835120b6115747db8bd40097a13e6915 because the tablet was not found. No further retry: Not found: Tablet not found: 835120b6115747db8bd40097a13e6915
I20260620 18:45:05.286856 23941 tablet_bootstrap.cc:492] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: Bootstrap starting.
I20260620 18:45:05.287541 23488 tablet_service.cc:1467] Processing CreateTablet for tablet 3636f6bdeab648508116ec4ebfe2c85f (DEFAULT_TABLE table=test-table [id=2e14f73d1aa149cc9bad7d7d3f7496e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:05.288118 23488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3636f6bdeab648508116ec4ebfe2c85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:05.288682 23941 tablet_bootstrap.cc:654] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:05.304533 23939 tablet_bootstrap.cc:492] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: Bootstrap starting.
I20260620 18:45:05.306670 23939 tablet_bootstrap.cc:654] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:05.314662 23863 tablet_bootstrap.cc:492] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: Bootstrap starting.
I20260620 18:45:05.316813 23863 tablet_bootstrap.cc:654] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:05.335065 23863 tablet_bootstrap.cc:492] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: No bootstrap required, opened a new log
I20260620 18:45:05.335320 23863 ts_tablet_manager.cc:1397] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.005s
I20260620 18:45:05.336387 23863 raft_consensus.cc:348] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.336715 23863 raft_consensus.cc:374] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:05.336813 23863 raft_consensus.cc:729] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa02816cd3143e19c0ca41888ca44ed, State: Initialized, Role: FOLLOWER
I20260620 18:45:05.337147 23863 consensus_queue.cc:260] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.340924 23863 ts_tablet_manager.cc:1428] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 18:45:05.346727 23939 tablet_bootstrap.cc:492] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: No bootstrap required, opened a new log
I20260620 18:45:05.346988 23939 ts_tablet_manager.cc:1397] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.012s
I20260620 18:45:05.348008 23939 raft_consensus.cc:348] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.348392 23939 raft_consensus.cc:374] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:05.348492 23939 raft_consensus.cc:729] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bce56caa1f43e09c9ba1d857c1d24f, State: Initialized, Role: FOLLOWER
I20260620 18:45:05.348824 23939 consensus_queue.cc:260] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.354031 23939 ts_tablet_manager.cc:1428] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260620 18:45:05.358238 23941 tablet_bootstrap.cc:492] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: No bootstrap required, opened a new log
I20260620 18:45:05.358469 23941 ts_tablet_manager.cc:1397] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: Time spent bootstrapping tablet: real 0.072s user 0.036s sys 0.034s
I20260620 18:45:05.359402 23941 raft_consensus.cc:348] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.359694 23941 raft_consensus.cc:374] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:05.359808 23941 raft_consensus.cc:729] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606e725f5dbb4697bf8a1acae5b97617, State: Initialized, Role: FOLLOWER
I20260620 18:45:05.360116 23941 consensus_queue.cc:260] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.360708 23941 ts_tablet_manager.cc:1428] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 18:45:05.437669 23914 raft_consensus.cc:482] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:05.437868 23914 raft_consensus.cc:504] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.438568 23914 leader_election.cc:288] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583), 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443)
I20260620 18:45:05.439080 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3636f6bdeab648508116ec4ebfe2c85f" candidate_uuid: "0aa02816cd3143e19c0ca41888ca44ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606e725f5dbb4697bf8a1acae5b97617" is_pre_election: true
I20260620 18:45:05.439141 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3636f6bdeab648508116ec4ebfe2c85f" candidate_uuid: "0aa02816cd3143e19c0ca41888ca44ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" is_pre_election: true
I20260620 18:45:05.439402 23649 raft_consensus.cc:2454] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa02816cd3143e19c0ca41888ca44ed in term 0.
I20260620 18:45:05.439435 23509 raft_consensus.cc:2454] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa02816cd3143e19c0ca41888ca44ed in term 0.
I20260620 18:45:05.439900 23720 leader_election.cc:302] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa02816cd3143e19c0ca41888ca44ed, b9bce56caa1f43e09c9ba1d857c1d24f; no voters:
I20260620 18:45:05.440181 23914 raft_consensus.cc:2781] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:05.440323 23914 raft_consensus.cc:482] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:05.440424 23914 raft_consensus.cc:3037] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:05.442318 23914 raft_consensus.cc:504] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.442838 23914 leader_election.cc:288] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [CANDIDATE]: Term 1 election: Requested vote from peers b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583), 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443)
I20260620 18:45:05.443357 23649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3636f6bdeab648508116ec4ebfe2c85f" candidate_uuid: "0aa02816cd3143e19c0ca41888ca44ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "606e725f5dbb4697bf8a1acae5b97617"
I20260620 18:45:05.443343 23509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3636f6bdeab648508116ec4ebfe2c85f" candidate_uuid: "0aa02816cd3143e19c0ca41888ca44ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f"
I20260620 18:45:05.443545 23509 raft_consensus.cc:3037] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:05.443570 23649 raft_consensus.cc:3037] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:05.445657 23649 raft_consensus.cc:2454] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa02816cd3143e19c0ca41888ca44ed in term 1.
I20260620 18:45:05.446244 23719 leader_election.cc:302] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa02816cd3143e19c0ca41888ca44ed, 606e725f5dbb4697bf8a1acae5b97617; no voters:
I20260620 18:45:05.446528 23914 raft_consensus.cc:2781] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:05.446614 23509 raft_consensus.cc:2454] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa02816cd3143e19c0ca41888ca44ed in term 1.
I20260620 18:45:05.447075 23914 raft_consensus.cc:686] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 LEADER]: Becoming Leader. State: Replica: 0aa02816cd3143e19c0ca41888ca44ed, State: Running, Role: LEADER
I20260620 18:45:05.447392 23914 consensus_queue.cc:237] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } }
I20260620 18:45:05.449952 23352 catalog_manager.cc:5495] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195). New cstate: current_term: 1 leader_uuid: "0aa02816cd3143e19c0ca41888ca44ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aa02816cd3143e19c0ca41888ca44ed" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 36643 } health_report { overall_health: HEALTHY } } }
I20260620 18:45:05.481761 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.475653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.15.102.195:36643)
0620 18:45:05.476554 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.476562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.476581 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:45:05.476645 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.476649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.476751 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.477346 (+ 595us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:05.477514 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.478562 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.480041 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.480448 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.480582 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.480993 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:05.481104 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:05.481204 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:05.481355 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:45:05.481430 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":659,"threads_started":1}
I20260620 18:45:05.495427 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.489840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:45273 (local address 127.15.102.195:36643)
0620 18:45:05.490382 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.490388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.490494 (+ 106us) server_negotiation.cc:407] Connection header received
0620 18:45:05.490687 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.490692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.490759 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.490847 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:05.491293 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.492019 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.493514 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.494142 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.494265 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.494694 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:05.494808 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:05.494889 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:05.494989 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:45:05.495061 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455}
I20260620 18:45:05.495702 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.489811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:36643 (local address 127.15.102.195:45273)
0620 18:45:05.490395 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:05.490413 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:05.490564 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:05.490861 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:05.490865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:05.490874 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 18:45:05.491124 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:05.491132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.492161 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.492165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:05.493364 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:05.493374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.494299 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.494305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:05.494476 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.494521 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 18:45:05.494899 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 18:45:05.494907 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:45:05.494969 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 18:45:05.495065 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":22,"thread_start_us":318,"threads_started":1}
I20260620 18:45:05.504009 23509 raft_consensus.cc:1261] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Refusing update from remote peer 0aa02816cd3143e19c0ca41888ca44ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:05.504206 23649 raft_consensus.cc:1261] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Refusing update from remote peer 0aa02816cd3143e19c0ca41888ca44ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:05.504561 23914 consensus_queue.cc:1035] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bce56caa1f43e09c9ba1d857c1d24f" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:05.504848 23954 consensus_queue.cc:1035] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "606e725f5dbb4697bf8a1acae5b97617" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:05.569808 23352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:05.570077 23352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:05.573171 23352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 601adf04640f4db98315eb65e3761506: Sending DeleteTablet for 3 replicas of tablet 3636f6bdeab648508116ec4ebfe2c85f
I20260620 18:45:05.573998 23488 tablet_service.cc:1514] Processing DeleteTablet for tablet 3636f6bdeab648508116ec4ebfe2c85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:05 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:49444
I20260620 18:45:05.574034 23628 tablet_service.cc:1514] Processing DeleteTablet for tablet 3636f6bdeab648508116ec4ebfe2c85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:05 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:58446
I20260620 18:45:05.574210 23768 tablet_service.cc:1514] Processing DeleteTablet for tablet 3636f6bdeab648508116ec4ebfe2c85f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:05 UTC) from {username='oryx', principal='oryx/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:52894
I20260620 18:45:05.574302 23924 tablet_replica.cc:330] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: stopping tablet replica
I20260620 18:45:05.574302 23926 tablet_replica.cc:330] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: stopping tablet replica
I20260620 18:45:05.574481 23924 raft_consensus.cc:2229] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:05.574483 23926 raft_consensus.cc:2229] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:05.574530 23925 tablet_replica.cc:330] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: stopping tablet replica
I20260620 18:45:05.574679 23924 raft_consensus.cc:2258] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:05.574679 23926 raft_consensus.cc:2258] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:05.574719 23925 raft_consensus.cc:2229] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:05.575268 23925 raft_consensus.cc:2258] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:05.576035 23926 ts_tablet_manager.cc:1905] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:05.576035 23924 ts_tablet_manager.cc:1905] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:05.576560 23925 ts_tablet_manager.cc:1905] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:05.579469 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 23423
I20260620 18:45:05.582726 23924 ts_tablet_manager.cc:1918] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:45:05.582867 23924 log.cc:1196] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/3636f6bdeab648508116ec4ebfe2c85f
I20260620 18:45:05.583294 23924 ts_tablet_manager.cc:1939] T 3636f6bdeab648508116ec4ebfe2c85f P b9bce56caa1f43e09c9ba1d857c1d24f: Deleting consensus metadata
I20260620 18:45:05.584622 23926 ts_tablet_manager.cc:1918] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:45:05.584645 23337 catalog_manager.cc:4841] TS b9bce56caa1f43e09c9ba1d857c1d24f (127.15.102.193:36583): tablet 3636f6bdeab648508116ec4ebfe2c85f (table test-table [id=2e14f73d1aa149cc9bad7d7d3f7496e5]) successfully deleted
I20260620 18:45:05.584875 23926 log.cc:1196] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/3636f6bdeab648508116ec4ebfe2c85f
I20260620 18:45:05.585363 23926 ts_tablet_manager.cc:1939] T 3636f6bdeab648508116ec4ebfe2c85f P 606e725f5dbb4697bf8a1acae5b97617: Deleting consensus metadata
I20260620 18:45:05.586440 23336 catalog_manager.cc:4841] TS 606e725f5dbb4697bf8a1acae5b97617 (127.15.102.194:35443): tablet 3636f6bdeab648508116ec4ebfe2c85f (table test-table [id=2e14f73d1aa149cc9bad7d7d3f7496e5]) successfully deleted
I20260620 18:45:05.589920 23925 ts_tablet_manager.cc:1918] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 18:45:05.590073 23925 log.cc:1196] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/3636f6bdeab648508116ec4ebfe2c85f
I20260620 18:45:05.590498 23925 ts_tablet_manager.cc:1939] T 3636f6bdeab648508116ec4ebfe2c85f P 0aa02816cd3143e19c0ca41888ca44ed: Deleting consensus metadata
I20260620 18:45:05.591624 23338 catalog_manager.cc:4841] TS 0aa02816cd3143e19c0ca41888ca44ed (127.15.102.195:36643): tablet 3636f6bdeab648508116ec4ebfe2c85f (table test-table [id=2e14f73d1aa149cc9bad7d7d3f7496e5]) successfully deleted
Jun 20 18:45:05 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.193@KRBTEST.COM
I20260620 18:45:05.602228 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.580814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.15.102.193:36583)
0620 18:45:05.584869 (+ 4055us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.584876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.585607 (+ 731us) server_negotiation.cc:407] Connection header received
0620 18:45:05.585684 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.585689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.585766 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.585870 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:05.587613 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.588782 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.590657 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.591277 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.591417 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.596411 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:05.596426 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:05.596444 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:05.596474 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:05.599633 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.600263 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.600270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.600273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.600382 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.600758 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.600763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.600766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.601091 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:05.601396 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:05.601648 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 18:45:05.601871 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3920,"thread_start_us":4110,"threads_started":1}
I20260620 18:45:05.602869 23554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:05.749439 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 23423
I20260620 18:45:05.780153 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 23563
Jun 20 18:45:05 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.194@KRBTEST.COM
I20260620 18:45:05.797921 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.780917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.15.102.194:35443)
0620 18:45:05.781381 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.781385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.781401 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:45:05.781454 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.781457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.781514 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.781602 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:05.783280 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.784504 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.786357 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.786754 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.786849 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.791847 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:05.791861 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:05.791877 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:05.791916 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:05.794818 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.795665 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.795672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.795675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.795813 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.796180 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.796187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.796193 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.796626 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:05.796937 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:05.797272 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 18:45:05.797484 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":271,"threads_started":1}
I20260620 18:45:05.798789 23694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:05.922683 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 23563
I20260620 18:45:05.953123 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 23703
Jun 20 18:45:05 dist-test-slave-zfp8 krb5kdc[23307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.102.195@KRBTEST.COM
I20260620 18:45:05.970613 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:05.954031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.15.102.195:36643)
0620 18:45:05.954205 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:05.954209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:05.954226 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:45:05.954334 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:05.954337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:05.954397 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:05.954479 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:05.956153 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.956951 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.958722 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:05.959162 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:05.959336 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:05.964455 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:05.964471 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:05.964487 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:05.964521 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:05.967889 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.968554 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.968563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.968565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.968701 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:05.969068 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:05.969074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:05.969077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:05.969419 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:05.969744 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:05.970037 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 18:45:05.970263 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:45:05.971338 23833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:06.126458 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 23703
I20260620 18:45:06.151399 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 23320
I20260620 18:45:06.152568 23392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:06.267181 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 23320
2026-06-20T18:45:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5531 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:45:06 dist-test-slave-zfp8 krb5kdc[23973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:45:06 dist-test-slave-zfp8 krb5kdc[23973](info): set up 2 sockets
Jun 20 18:45:06 dist-test-slave-zfp8 krb5kdc[23973](info): commencing operation
krb5kdc: starting...
W20260620 18:45:07.372417 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:45:07 dist-test-slave-zfp8 krb5kdc[23973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:45:07.591540 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37241
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:37241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:07.911826 23986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:07.912189 23986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:07.912356 23986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:07.919823 23986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:07.919983 23986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:07.920092 23986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:07.920203 23986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:07.920295 23986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:07.930224 23986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:37241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:07.931879 23986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:07.933501 23986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:07.945250 23991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:07.945355 23992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:07.945392 23994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:07.946421 23986 server_base.cc:1034] running on GCE node
I20260620 18:45:07.947270 23986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:07.948633 23986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:07.949826 23986 hybrid_clock.cc:648] HybridClock initialized: now 1781981107949770 us; error 57 us; skew 500 ppm
I20260620 18:45:07.950233 23986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T18:45:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1750 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:45:08 dist-test-slave-zfp8 krb5kdc[24000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:45:08 dist-test-slave-zfp8 krb5kdc[24000](info): set up 2 sockets
Jun 20 18:45:08 dist-test-slave-zfp8 krb5kdc[24000](info): commencing operation
krb5kdc: starting...
W20260620 18:45:09.116426 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:45:09 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:45:09.332736 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40393
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:40393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:09.651305 24013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:09.651681 24013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:09.651857 24013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:09.658999 24013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:09.659161 24013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:09.659312 24013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:09.659411 24013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:09.659500 24013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:09.669472 24013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:40393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:09.670862 24013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:09.672196 24013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:09.683573 24021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:09.683606 24018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:09.683640 24019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:09.684844 24013 server_base.cc:1034] running on GCE node
I20260620 18:45:09.685595 24013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:09.686954 24013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:09.688148 24013 hybrid_clock.cc:648] HybridClock initialized: now 1781981109688094 us; error 55 us; skew 500 ppm
I20260620 18:45:09.688444 24013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:09 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981109, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:09.691563 24013 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:45:09.693111 24013 webserver.cc:460] Webserver started at http://127.15.102.254:42859/ using document root <none> and password file <none>
I20260620 18:45:09.693859 24013 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:09.694005 24013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:09.694346 24013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:09.697511 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a"
format_stamp: "Formatted at 2026-06-20 18:45:09 on dist-test-slave-zfp8"
server_key: "d5db091f0157533adb7947e4308d7ec2"
server_key_iv: "3d10a0deef3405ab0e9beb4a0dbe518b"
server_key_version: "encryptionkey@0"
I20260620 18:45:09.698328 24013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a"
format_stamp: "Formatted at 2026-06-20 18:45:09 on dist-test-slave-zfp8"
server_key: "d5db091f0157533adb7947e4308d7ec2"
server_key_iv: "3d10a0deef3405ab0e9beb4a0dbe518b"
server_key_version: "encryptionkey@0"
I20260620 18:45:09.703814 24013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:45:09.707428 24028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:09.708740 24013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:09.708933 24013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a"
format_stamp: "Formatted at 2026-06-20 18:45:09 on dist-test-slave-zfp8"
server_key: "d5db091f0157533adb7947e4308d7ec2"
server_key_iv: "3d10a0deef3405ab0e9beb4a0dbe518b"
server_key_version: "encryptionkey@0"
I20260620 18:45:09.709146 24013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:09.738960 24013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:09.743861 24013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:09.744184 24013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:09.763680 24013 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:40393
I20260620 18:45:09.763725 24079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:40393 every 8 connection(s)
I20260620 18:45:09.765148 24013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:09.768932 24080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:09.773900 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 24013
I20260620 18:45:09.774115 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:45:09.774904 15771 external_mini_cluster.cc:1428] Setting key fff123352b7d7910f1536dce1aa754e8
I20260620 18:45:09.780129 24080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Bootstrap starting.
I20260620 18:45:09.783808 24080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:09.785006 24080 log.cc:824] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:09.788302 24080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: No bootstrap required, opened a new log
Jun 20 18:45:09 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:45:09.796264 24080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } }
I20260620 18:45:09.796677 24080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:09.796792 24080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8121b0cef30d4112aaeb8e7f8dd7ae7a, State: Initialized, Role: FOLLOWER
I20260620 18:45:09.797428 24080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } }
I20260620 18:45:09.797683 24080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:09.797842 24080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:09.798035 24080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:09.799465 24083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:09.777100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.15.102.254:40393)
0620 18:45:09.778041 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:09.778054 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:09.778111 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:45:09.780016 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:09.780040 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:09.780493 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:09.780837 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:09.782635 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:09.784187 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:09.785922 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:09.786802 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:09.786947 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:09.791530 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:09.791537 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:09.791553 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:09.791599 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:09.794971 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:09.795765 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:09.795772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:09.795778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:09.795946 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:09.796347 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:09.796354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:09.796358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:09.797065 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:09.797389 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:09.797844 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 18:45:09.798272 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":456,"threads_started":1}
I20260620 18:45:09.800905 24080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } }
I20260620 18:45:09.801398 24080 leader_election.cc:302] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8121b0cef30d4112aaeb8e7f8dd7ae7a; no voters:
I20260620 18:45:09.802101 24080 leader_election.cc:288] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:45:09.802205 24085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:09.803274 24085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 1 LEADER]: Becoming Leader. State: Replica: 8121b0cef30d4112aaeb8e7f8dd7ae7a, State: Running, Role: LEADER
I20260620 18:45:09.803807 24085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } }
I20260620 18:45:09.804334 24080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:09.806633 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } } }
I20260620 18:45:09.807022 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:09.807672 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8121b0cef30d4112aaeb8e7f8dd7ae7a. Latest consensus state: current_term: 1 leader_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } } }
I20260620 18:45:09.808135 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:09.810582 24091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:09.816675 24091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:09.826076 24091 catalog_manager.cc:1347] Generated new cluster ID: 88de055ff03f4e738d2c5377e7630382
I20260620 18:45:09.826211 24091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:09.834110 24091 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:45:09.835611 24091 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:09.841919 24091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Generated new TSK 0
I20260620 18:45:09.842813 24091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:45:09.894476 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40393
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:10.226482 24106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:10.226837 24106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:10.227006 24106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:10.234450 24106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:10.234609 24106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:10.234799 24106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:45:10.244977 24106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:10.246390 24106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:10.247848 24106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:10.261395 24114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:10.261489 24112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:10.261464 24111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:10.262395 24106 server_base.cc:1034] running on GCE node
I20260620 18:45:10.263273 24106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:10.264293 24106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:10.265502 24106 hybrid_clock.cc:648] HybridClock initialized: now 1781981110265428 us; error 67 us; skew 500 ppm
I20260620 18:45:10.265848 24106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:10 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:10.273417 24106 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:45:10.274960 24106 webserver.cc:460] Webserver started at http://127.15.102.193:34663/ using document root <none> and password file <none>
I20260620 18:45:10.275734 24106 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:10.275843 24106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:10.276118 24106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:10.279094 24106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "36a5c9f7351648f0be74b5eaa5414ee2"
format_stamp: "Formatted at 2026-06-20 18:45:10 on dist-test-slave-zfp8"
server_key: "5cfc9c473e3cbb3b9c3433b4e1d2bc4c"
server_key_iv: "69e89c5bf1402e65a039d7645ed451b0"
server_key_version: "encryptionkey@0"
I20260620 18:45:10.279963 24106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "36a5c9f7351648f0be74b5eaa5414ee2"
format_stamp: "Formatted at 2026-06-20 18:45:10 on dist-test-slave-zfp8"
server_key: "5cfc9c473e3cbb3b9c3433b4e1d2bc4c"
server_key_iv: "69e89c5bf1402e65a039d7645ed451b0"
server_key_version: "encryptionkey@0"
I20260620 18:45:10.285172 24106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:45:10.288744 24121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.290083 24106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:45:10.290283 24106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "36a5c9f7351648f0be74b5eaa5414ee2"
format_stamp: "Formatted at 2026-06-20 18:45:10 on dist-test-slave-zfp8"
server_key: "5cfc9c473e3cbb3b9c3433b4e1d2bc4c"
server_key_iv: "69e89c5bf1402e65a039d7645ed451b0"
server_key_version: "encryptionkey@0"
I20260620 18:45:10.290483 24106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:10.313701 24106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:10.317783 24106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:10.318092 24106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:10.319092 24106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:10.320430 24106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:10.320546 24106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.320694 24106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:10.320763 24106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.352922 24106 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:38639
I20260620 18:45:10.352963 24233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:38639 every 8 connection(s)
I20260620 18:45:10.354214 24106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:10.359822 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 24106
I20260620 18:45:10.360054 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:45:10.360925 15771 external_mini_cluster.cc:1428] Setting key 76d6b66d14169111b61e199ecbf89666
Jun 20 18:45:10 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:45:10.378283 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.357051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:53533 (local address 127.15.102.254:40393)
0620 18:45:10.357528 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:10.357534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:10.358548 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 18:45:10.361284 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:10.361289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:10.361357 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:10.361452 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:10.363540 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:10.364253 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:10.367039 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:10.367487 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:10.367604 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:10.372082 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:10.372089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:10.372094 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:10.372140 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:10.374773 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:10.375880 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:10.375884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:10.375886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:10.375998 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:10.376502 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:10.376505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:10.376507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:10.376768 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:10.376874 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:10.377864 (+ 990us) server_negotiation.cc:299] Negotiation successful
0620 18:45:10.378028 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260620 18:45:10.380117 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.357620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:53533)
0620 18:45:10.358370 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:10.358417 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:10.360949 (+ 2532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:10.361764 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:10.361774 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:10.362486 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:10.363341 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:10.363368 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:10.364434 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:10.364444 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:10.366867 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:10.366886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:10.367723 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:10.367733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:10.368084 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:10.368855 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:10.368892 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:10.371833 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:10.374936 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:10.374950 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:10.374973 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:10.375740 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:10.376114 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:10.376123 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:10.376130 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:10.376388 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:10.376912 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:10.376925 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:10.377437 (+ 512us) client_negotiation.cc:769] Sending connection context
0620 18:45:10.377769 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 18:45:10.378405 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":300,"threads_started":1}
I20260620 18:45:10.382005 24234 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40393
I20260620 18:45:10.382349 24234 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:10.383121 24234 heartbeater.cc:511] Master 127.15.102.254:40393 requested a full tablet report, sending...
I20260620 18:45:10.385731 24045 ts_manager.cc:194] Registered new tserver with Master: 36a5c9f7351648f0be74b5eaa5414ee2 (127.15.102.193:38639)
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:45:10.387676 24045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:53533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:45:10.417093 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40393
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:10.758796 24240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:10.759097 24240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:10.759321 24240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:10.766908 24240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:10.767019 24240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:10.767197 24240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:45:10.777256 24240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:10.778486 24240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:10.779856 24240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:10.793303 24245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:10.793257 24248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:10.793331 24246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:10.794237 24240 server_base.cc:1034] running on GCE node
I20260620 18:45:10.794924 24240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:10.795912 24240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:10.797142 24240 hybrid_clock.cc:648] HybridClock initialized: now 1781981110797090 us; error 48 us; skew 500 ppm
I20260620 18:45:10.797459 24240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:10 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:10.800613 24240 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:45:10.802114 24240 webserver.cc:460] Webserver started at http://127.15.102.194:33901/ using document root <none> and password file <none>
I20260620 18:45:10.802824 24240 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:10.802964 24240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:10.803318 24240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:10.806474 24240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "4662b7db946f4c41af40227bd5dded6a"
format_stamp: "Formatted at 2026-06-20 18:45:10 on dist-test-slave-zfp8"
server_key: "244ff814303430a4e8e9b408fdf996e7"
server_key_iv: "62e19f12c0b5ed7ab720e21b0016be0a"
server_key_version: "encryptionkey@0"
I20260620 18:45:10.807341 24240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "4662b7db946f4c41af40227bd5dded6a"
format_stamp: "Formatted at 2026-06-20 18:45:10 on dist-test-slave-zfp8"
server_key: "244ff814303430a4e8e9b408fdf996e7"
server_key_iv: "62e19f12c0b5ed7ab720e21b0016be0a"
server_key_version: "encryptionkey@0"
I20260620 18:45:10.812569 24240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:45:10.816148 24255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.817534 24240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:10.817729 24240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "4662b7db946f4c41af40227bd5dded6a"
format_stamp: "Formatted at 2026-06-20 18:45:10 on dist-test-slave-zfp8"
server_key: "244ff814303430a4e8e9b408fdf996e7"
server_key_iv: "62e19f12c0b5ed7ab720e21b0016be0a"
server_key_version: "encryptionkey@0"
I20260620 18:45:10.817934 24240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:10.834482 24240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:10.838699 24240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:10.839028 24240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:10.840140 24240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:10.841569 24240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:10.841681 24240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.841845 24240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:10.841929 24240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:10.877764 24240 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:34951
I20260620 18:45:10.877799 24367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:34951 every 8 connection(s)
I20260620 18:45:10.879220 24240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:10.889199 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 24240
I20260620 18:45:10.889392 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:45:10.889876 15771 external_mini_cluster.cc:1428] Setting key 0e65d23e1a1e1a8ec2c39e22d7d3bccd
Jun 20 18:45:10 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981110, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:45:10.899832 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.881952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:49095 (local address 127.15.102.254:40393)
0620 18:45:10.882387 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:10.882391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:10.883416 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 18:45:10.885277 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:10.885282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:10.885374 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:10.885503 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:10.886985 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:10.887761 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:10.889628 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:10.890113 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:10.890218 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:10.894122 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:10.894126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:10.894130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:10.894159 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:10.896811 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:10.897534 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:10.897541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:10.897544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:10.897677 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:10.898100 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:10.898107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:10.898111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:10.898363 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:10.898522 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:10.899202 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 18:45:10.899448 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":245,"threads_started":1}
I20260620 18:45:10.900812 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:10.882489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:49095)
0620 18:45:10.883200 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:10.883299 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:10.885005 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:10.885750 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:10.885758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:10.886259 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:10.886818 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:10.886835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:10.887912 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:10.887922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:10.889472 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:10.889487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:10.890339 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:10.890347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:10.890607 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:10.891434 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:10.891456 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:10.893839 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:10.896963 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:10.896972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:10.896991 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:10.897411 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:10.897787 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:10.897792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:10.897796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:10.897985 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:10.898490 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:10.898497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:10.898842 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 18:45:10.899127 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 18:45:10.899687 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":282,"threads_started":1}
I20260620 18:45:10.902213 24368 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40393
I20260620 18:45:10.902516 24368 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:10.903224 24368 heartbeater.cc:511] Master 127.15.102.254:40393 requested a full tablet report, sending...
I20260620 18:45:10.904809 24045 ts_manager.cc:194] Registered new tserver with Master: 4662b7db946f4c41af40227bd5dded6a (127.15.102.194:34951)
I20260620 18:45:10.906157 24045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:49095
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:45:10.938519 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40393
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:11.260810 24374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:11.261090 24374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:11.261191 24374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:11.268455 24374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:11.268560 24374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:11.268671 24374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:45:11.278771 24374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:11.280212 24374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:11.281901 24374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:11.295585 24380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:11.295667 24382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:11.296307 24379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:11.296720 24374 server_base.cc:1034] running on GCE node
I20260620 18:45:11.297363 24374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:11.298352 24374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:11.299630 24374 hybrid_clock.cc:648] HybridClock initialized: now 1781981111299582 us; error 43 us; skew 500 ppm
I20260620 18:45:11.299978 24374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:11 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981111, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:11.303064 24374 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:45:11.304562 24374 webserver.cc:460] Webserver started at http://127.15.102.195:45757/ using document root <none> and password file <none>
I20260620 18:45:11.305305 24374 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:11.305437 24374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:11.305737 24374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:11.308737 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "99554700500c4ae5b2fa0eef54813bfe"
format_stamp: "Formatted at 2026-06-20 18:45:11 on dist-test-slave-zfp8"
server_key: "7606131e96a5135cbc901e43e4e12ca4"
server_key_iv: "c8f9d861a15d14ec81259441372d1368"
server_key_version: "encryptionkey@0"
I20260620 18:45:11.309528 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "99554700500c4ae5b2fa0eef54813bfe"
format_stamp: "Formatted at 2026-06-20 18:45:11 on dist-test-slave-zfp8"
server_key: "7606131e96a5135cbc901e43e4e12ca4"
server_key_iv: "c8f9d861a15d14ec81259441372d1368"
server_key_version: "encryptionkey@0"
I20260620 18:45:11.314752 24374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:45:11.318413 24389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:11.319784 24374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:11.319954 24374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "99554700500c4ae5b2fa0eef54813bfe"
format_stamp: "Formatted at 2026-06-20 18:45:11 on dist-test-slave-zfp8"
server_key: "7606131e96a5135cbc901e43e4e12ca4"
server_key_iv: "c8f9d861a15d14ec81259441372d1368"
server_key_version: "encryptionkey@0"
I20260620 18:45:11.320171 24374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:11.354312 24374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:11.358457 24374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:11.358763 24374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:11.359766 24374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:11.361063 24374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:11.361168 24374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:11.361320 24374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:11.361389 24374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:11.390718 24234 heartbeater.cc:503] Master 127.15.102.254:40393 was elected leader, sending a full tablet report...
I20260620 18:45:11.394842 24374 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:45031
I20260620 18:45:11.394930 24501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:45031 every 8 connection(s)
I20260620 18:45:11.396196 24374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:11.400063 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 24374
I20260620 18:45:11.400264 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:45:11.400815 15771 external_mini_cluster.cc:1428] Setting key 5c2c3934bc8f397696ba3469cecb068e
Jun 20 18:45:11 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981111, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:45:11.418134 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:11.398642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:53611 (local address 127.15.102.254:40393)
0620 18:45:11.398820 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:11.398824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:11.400268 (+ 1444us) server_negotiation.cc:407] Connection header received
0620 18:45:11.402714 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:11.402720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:11.402812 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:11.402920 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:11.404897 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.406164 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.408601 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.409027 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.409129 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:11.412466 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:11.412471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:11.412474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:11.412506 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:11.415435 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:11.416169 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:11.416173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:11.416175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:11.416341 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:11.416732 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:11.416735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:11.416737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:11.416991 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:11.417101 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:11.417741 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 18:45:11.417901 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:45:11.419175 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:11.399223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:53611)
0620 18:45:11.400090 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:11.400146 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:11.402406 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:11.403612 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:11.403625 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:11.404147 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:11.404748 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:11.404767 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.406344 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.406350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:11.408427 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:11.408462 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.409221 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.409227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:11.409419 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:11.410033 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:11.410069 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:11.412215 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:11.415586 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:11.415595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:11.415612 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:11.416072 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:11.416438 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:11.416442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:11.416444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:11.416623 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:11.417092 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:11.417098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:11.417408 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 18:45:11.417642 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 18:45:11.418011 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":316,"threads_started":1}
I20260620 18:45:11.420598 24502 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40393
I20260620 18:45:11.420917 24502 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:11.421675 24502 heartbeater.cc:511] Master 127.15.102.254:40393 requested a full tablet report, sending...
I20260620 18:45:11.423089 24045 ts_manager.cc:194] Registered new tserver with Master: 99554700500c4ae5b2fa0eef54813bfe (127.15.102.195:45031)
I20260620 18:45:11.423920 24045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:53611
I20260620 18:45:11.428339 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab.
Entry for principal oryx/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab.
WARNING: no policy specified for oryx/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab.
Entry for principal oryx/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab.
WARNING: no policy specified for oryx/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab.
Entry for principal oryx/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab.
I20260620 18:45:11.558722 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24106
Jun 20 18:45:11 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:45:11.574633 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:11.559939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.15.102.193:38639)
0620 18:45:11.560632 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:11.560639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:11.560665 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:45:11.560727 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:11.560733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:11.560950 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:11.561093 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:11.562437 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.563529 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.564768 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.565315 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.565410 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:11.569294 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:11.569305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:11.569320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:11.569359 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:11.572205 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:11.572850 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:11.572858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:11.572860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:11.572977 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:11.573265 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:11.573269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:11.573271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:11.573594 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:11.573808 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:11.574039 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:45:11.574233 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":383,"threads_started":1}
I20260620 18:45:11.575968 24229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:11.658124 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 24106
I20260620 18:45:11.674779 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24240
Jun 20 18:45:11 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:45:11.690254 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:11.675703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.15.102.194:34951)
0620 18:45:11.676259 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:11.676267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:11.676291 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:45:11.676351 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:11.676357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:11.676564 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:11.676698 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:11.677890 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.678919 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.680302 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.680866 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.680966 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:11.685002 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:11.685013 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:11.685028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:11.685063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:11.687865 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:11.688424 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:11.688431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:11.688433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:11.688564 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:11.688853 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:11.688856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:11.688858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:11.689166 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:11.689386 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:11.689627 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 18:45:11.689864 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":312,"threads_started":1}
I20260620 18:45:11.691566 24363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:11.778443 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 24240
I20260620 18:45:11.794112 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24374
Jun 20 18:45:11 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:45:11.810403 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:11.795222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.15.102.195:45031)
0620 18:45:11.795867 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:11.795873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:11.795902 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:45:11.795964 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:11.795971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:11.796187 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:11.796321 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:11.797558 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.798567 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.799901 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:11.800535 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:11.800652 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:11.804981 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:11.805005 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:11.805022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:11.805056 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:11.807960 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:11.808650 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:11.808658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:11.808660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:11.808799 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:11.809106 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:11.809109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:11.809111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:11.809425 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:11.809735 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:11.809918 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 18:45:11.810113 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":317,"threads_started":1}
I20260620 18:45:11.811863 24497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:11.894161 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 24374
I20260620 18:45:11.910209 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24013
I20260620 18:45:11.911439 24075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:12.012109 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 24013
I20260620 18:45:12.032728 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40393
--webserver_interface=127.15.102.254
--webserver_port=42859
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:40393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:12.353842 24519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:12.354184 24519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:12.354357 24519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:12.361876 24519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:12.362033 24519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:12.362154 24519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:12.362251 24519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:12.362344 24519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:12.372332 24519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:40393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:12.373757 24519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:12.375092 24519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:12.387131 24525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:12.387199 24524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:12.387176 24527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:12.387483 24519 server_base.cc:1034] running on GCE node
I20260620 18:45:12.388513 24519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:12.389953 24519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:12.391177 24519 hybrid_clock.cc:648] HybridClock initialized: now 1781981112391107 us; error 63 us; skew 500 ppm
I20260620 18:45:12.391535 24519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:12 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981112, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:12.394542 24519 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:45:12.396044 24519 webserver.cc:460] Webserver started at http://127.15.102.254:42859/ using document root <none> and password file <none>
I20260620 18:45:12.396740 24519 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:12.396893 24519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:12.402213 24519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 18:45:12.405164 24534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:12.406520 24519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:45:12.406708 24519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a"
format_stamp: "Formatted at 2026-06-20 18:45:09 on dist-test-slave-zfp8"
server_key: "d5db091f0157533adb7947e4308d7ec2"
server_key_iv: "3d10a0deef3405ab0e9beb4a0dbe518b"
server_key_version: "encryptionkey@0"
I20260620 18:45:12.407744 24519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:12.426453 24519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:12.430706 24519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:12.430980 24519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:12.449532 24519 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:40393
I20260620 18:45:12.449549 24585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:40393 every 8 connection(s)
I20260620 18:45:12.451174 24519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:12.454246 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 24519
I20260620 18:45:12.455487 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:38639
--local_ip_for_outbound_sockets=127.15.102.193
--tserver_master_addrs=127.15.102.254:40393
--webserver_port=34663
--webserver_interface=127.15.102.193
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.102.193
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:45:12.462056 24586 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:45:12.466099 24586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Bootstrap starting.
I20260620 18:45:12.485687 24586 log.cc:824] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:12.494226 24586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:45:12.494764 24586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Bootstrap complete.
I20260620 18:45:12.503786 24586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } }
I20260620 18:45:12.504616 24586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8121b0cef30d4112aaeb8e7f8dd7ae7a, State: Initialized, Role: FOLLOWER
I20260620 18:45:12.505195 24586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } }
I20260620 18:45:12.505440 24586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:12.505568 24586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:12.505752 24586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:45:12.509480 24586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } }
I20260620 18:45:12.509981 24586 leader_election.cc:302] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8121b0cef30d4112aaeb8e7f8dd7ae7a; no voters:
I20260620 18:45:12.510658 24586 leader_election.cc:288] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:45:12.510788 24590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:45:12.511929 24590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [term 2 LEADER]: Becoming Leader. State: Replica: 8121b0cef30d4112aaeb8e7f8dd7ae7a, State: Running, Role: LEADER
I20260620 18:45:12.512712 24590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } }
I20260620 18:45:12.512972 24586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:12.515024 24592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8121b0cef30d4112aaeb8e7f8dd7ae7a. Latest consensus state: current_term: 2 leader_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } } }
I20260620 18:45:12.515357 24592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:12.515872 24591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8121b0cef30d4112aaeb8e7f8dd7ae7a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40393 } } }
I20260620 18:45:12.516161 24591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:12.528282 24597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:12.533680 24597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:12.534986 24597 catalog_manager.cc:1259] Loaded cluster ID: 88de055ff03f4e738d2c5377e7630382
I20260620 18:45:12.535077 24597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:12.538872 24597 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:12.540319 24597 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8121b0cef30d4112aaeb8e7f8dd7ae7a: Loaded TSK: 0
I20260620 18:45:12.541841 24597 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:45:12.799388 24588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:12.799696 24588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:12.799835 24588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:12.807812 24588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:12.807927 24588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:12.808099 24588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:45:12.818852 24588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.193.keytab
--principal=oryx/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:38639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=34663
--tserver_master_addrs=127.15.102.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:12.820334 24588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:12.821718 24588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:12.836663 24612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:12.836704 24613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:12.837885 24615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:12.838173 24588 server_base.cc:1034] running on GCE node
I20260620 18:45:12.838819 24588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:12.840008 24588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:12.841238 24588 hybrid_clock.cc:648] HybridClock initialized: now 1781981112841195 us; error 39 us; skew 500 ppm
I20260620 18:45:12.841593 24588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:12 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981112, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:12.845067 24588 init.cc:377] Logged in from keytab as oryx/127.15.102.193@KRBTEST.COM (short username oryx)
I20260620 18:45:12.846800 24588 webserver.cc:460] Webserver started at http://127.15.102.193:34663/ using document root <none> and password file <none>
I20260620 18:45:12.847674 24588 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:12.847838 24588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:12.854266 24588 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:45:12.857353 24622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:12.858913 24588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:45:12.859108 24588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "36a5c9f7351648f0be74b5eaa5414ee2"
format_stamp: "Formatted at 2026-06-20 18:45:10 on dist-test-slave-zfp8"
server_key: "5cfc9c473e3cbb3b9c3433b4e1d2bc4c"
server_key_iv: "69e89c5bf1402e65a039d7645ed451b0"
server_key_version: "encryptionkey@0"
I20260620 18:45:12.860265 24588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:12.882887 24588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:12.887163 24588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:12.887482 24588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:12.888494 24588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:12.889838 24588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:12.889967 24588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:12.890123 24588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:12.890192 24588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:12.923151 24588 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:38639
I20260620 18:45:12.923264 24734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:38639 every 8 connection(s)
I20260620 18:45:12.924623 24588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:12.931358 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 24588
I20260620 18:45:12.932066 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:34951
--local_ip_for_outbound_sockets=127.15.102.194
--tserver_master_addrs=127.15.102.254:40393
--webserver_port=33901
--webserver_interface=127.15.102.194
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.102.194
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:45:12 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:12 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:12.944319 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:12.927453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:45599)
0620 18:45:12.928244 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:12.928295 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:12.929975 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:12.931563 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:12.931574 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:12.932079 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:12.932722 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:12.932741 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.935108 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.935112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:12.936430 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:12.936441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.937939 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.937944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:12.938138 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:12.938842 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:12.938882 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:12.942987 (+ 4105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":329,"threads_started":1}
W20260620 18:45:12.944343 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:12.927392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45599 (local address 127.15.102.254:40393)
0620 18:45:12.928043 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:12.928054 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:12.928420 (+ 366us) server_negotiation.cc:407] Connection header received
0620 18:45:12.930496 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:12.930509 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:12.930940 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:12.931398 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:12.932926 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.934922 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.936576 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.937710 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.937915 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:12.943363 (+ 5448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:45599: BlockingRecv error: recv got EOF from 127.15.102.193:45599 (error 108)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":254,"threads_started":1}
W20260620 18:45:12.945061 24735 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:40393 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.102.254:40393: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Jun 20 18:45:12 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:12 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:12.953318 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:12.945658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:45643)
0620 18:45:12.945921 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:12.945959 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:12.946062 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:12.946426 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:12.946434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:12.946757 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:12.946890 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:12.946897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.947990 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.947995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:12.948869 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:12.948880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.949558 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.949563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:12.949738 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:12.950310 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:12.950327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:12.953019 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 18:45:12.953431 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:12.945812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45643 (local address 127.15.102.254:40393)
0620 18:45:12.945994 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:12.946001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:12.946017 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:45:12.946197 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:12.946203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:12.946267 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:12.946376 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:12.947028 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.947851 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.949009 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.949398 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.949535 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:12.953217 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:45643: BlockingRecv error: recv got EOF from 127.15.102.193:45643 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:45:12 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:12 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:12.961918 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:12.954091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:58055)
0620 18:45:12.954264 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:12.954284 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:12.954388 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:12.954713 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:12.954717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:12.955010 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:12.955151 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:12.955158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.956148 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.956153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:12.957042 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:12.957052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.957736 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.957741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:12.957906 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:12.958482 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:12.958498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:12.961622 (+ 3124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:45:12.962231 24735 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:45:12.962390 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:12.954152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:58055 (local address 127.15.102.254:40393)
0620 18:45:12.954302 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:12.954309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:12.954326 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:45:12.954502 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:12.954508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:12.954582 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:12.954691 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:12.955300 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.955999 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.957185 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:12.957581 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:12.957714 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:12.961834 (+ 4120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:58055: BlockingRecv error: recv got EOF from 127.15.102.193:58055 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 18:45:13.260635 24739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:13.260912 24739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:13.261086 24739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:13.268339 24739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:13.268443 24739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:13.268558 24739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:45:13.278206 24739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.194.keytab
--principal=oryx/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:34951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=33901
--tserver_master_addrs=127.15.102.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:13.279529 24739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:13.280823 24739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:13.293985 24744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:13.293972 24747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:13.293926 24745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:13.294648 24739 server_base.cc:1034] running on GCE node
I20260620 18:45:13.295352 24739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:13.296317 24739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:13.297518 24739 hybrid_clock.cc:648] HybridClock initialized: now 1781981113297461 us; error 61 us; skew 500 ppm
I20260620 18:45:13.297838 24739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981113, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:13.300987 24739 init.cc:377] Logged in from keytab as oryx/127.15.102.194@KRBTEST.COM (short username oryx)
I20260620 18:45:13.302416 24739 webserver.cc:460] Webserver started at http://127.15.102.194:33901/ using document root <none> and password file <none>
I20260620 18:45:13.303120 24739 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:13.303282 24739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:13.308583 24739 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:45:13.311313 24754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:13.312716 24739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:45:13.312882 24739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "4662b7db946f4c41af40227bd5dded6a"
format_stamp: "Formatted at 2026-06-20 18:45:10 on dist-test-slave-zfp8"
server_key: "244ff814303430a4e8e9b408fdf996e7"
server_key_iv: "62e19f12c0b5ed7ab720e21b0016be0a"
server_key_version: "encryptionkey@0"
I20260620 18:45:13.313851 24739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:13.327154 24739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:13.330921 24739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:13.331223 24739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:13.332194 24739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:13.333516 24739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:13.333639 24739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:13.333781 24739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:13.333870 24739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:13.366480 24739 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:34951
I20260620 18:45:13.366566 24866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:34951 every 8 connection(s)
I20260620 18:45:13.368033 24739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:13.377143 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 24739
I20260620 18:45:13.377758 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:45031
--local_ip_for_outbound_sockets=127.15.102.195
--tserver_master_addrs=127.15.102.254:40393
--webserver_port=45757
--webserver_interface=127.15.102.195
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.102.195
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:45:13.383689 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:13.370549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:54757 (local address 127.15.102.254:40393)
0620 18:45:13.370729 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:13.370734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:13.371973 (+ 1239us) server_negotiation.cc:407] Connection header received
0620 18:45:13.373806 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:13.373810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:13.373883 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:13.373987 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:13.375598 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.376306 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.377870 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.378369 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.378505 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.383473 (+ 4968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:54757: BlockingRecv error: recv got EOF from 127.15.102.194:54757 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:45:13.384590 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.371069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:54757)
0620 18:45:13.371789 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.371834 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:13.373530 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:13.374198 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:13.374206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:13.374803 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:13.375442 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.375461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.376432 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.376436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.377716 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.377727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.378599 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.378607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.378826 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.379538 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:13.379558 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:13.383195 (+ 3637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":316,"threads_started":1}
W20260620 18:45:13.385304 24867 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:40393 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.102.254:40393: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:13.394183 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.385939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:39643)
0620 18:45:13.386132 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.386153 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:13.386249 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:13.386652 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:13.386655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:13.386927 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:13.387055 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.387062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.388560 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.388563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.389419 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.389429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.390502 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.390505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.390619 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.391225 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:13.391286 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:13.393898 (+ 2612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 18:45:13.394354 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:13.386087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:39643 (local address 127.15.102.254:40393)
0620 18:45:13.386298 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:13.386307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:13.386331 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:45:13.386394 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:13.386403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:13.386501 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:13.386610 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:13.387203 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.388419 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.389565 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.390358 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.390520 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.394094 (+ 3574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:39643: BlockingRecv error: recv got EOF from 127.15.102.194:39643 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:13.402426 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.395140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:59161)
0620 18:45:13.395343 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.395362 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:13.395460 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:13.395777 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:13.395781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:13.396064 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:13.396197 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.396205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.397131 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.397136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.398025 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.398035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.398690 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.398696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.398883 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.399566 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:13.399583 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:13.402124 (+ 2541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:45:13.402534 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:13.395229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:59161 (local address 127.15.102.254:40393)
0620 18:45:13.395416 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:13.395421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:13.395438 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:45:13.395585 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:13.395590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:13.395656 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:13.395756 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:13.396327 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.397004 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.398170 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.398552 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.398687 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.402310 (+ 3623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:59161: BlockingRecv error: recv got EOF from 127.15.102.194:59161 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 18:45:13.402907 24867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:45:13.712795 24870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:13.713117 24870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:13.713279 24870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:13.721335 24870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:13.721444 24870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:13.721652 24870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:45:13.731462 24870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/krb5kdc/oryx_127.15.102.195.keytab
--principal=oryx/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:45031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=45757
--tserver_master_addrs=127.15.102.254:40393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:13.732831 24870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:13.734143 24870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:13.747454 24878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:13.747916 24875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:13.748184 24876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:13.748304 24870 server_base.cc:1034] running on GCE node
I20260620 18:45:13.749019 24870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:13.750062 24870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:13.751360 24870 hybrid_clock.cc:648] HybridClock initialized: now 1781981113751313 us; error 54 us; skew 500 ppm
I20260620 18:45:13.751717 24870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981113, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:13.754994 24870 init.cc:377] Logged in from keytab as oryx/127.15.102.195@KRBTEST.COM (short username oryx)
I20260620 18:45:13.756623 24870 webserver.cc:460] Webserver started at http://127.15.102.195:45757/ using document root <none> and password file <none>
I20260620 18:45:13.757337 24870 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:13.757524 24870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:13.763589 24870 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:45:13.766456 24885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:13.768008 24870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:13.768224 24870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "99554700500c4ae5b2fa0eef54813bfe"
format_stamp: "Formatted at 2026-06-20 18:45:11 on dist-test-slave-zfp8"
server_key: "7606131e96a5135cbc901e43e4e12ca4"
server_key_iv: "c8f9d861a15d14ec81259441372d1368"
server_key_version: "encryptionkey@0"
I20260620 18:45:13.769305 24870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:13.788403 24870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:13.793138 24870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:13.793469 24870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:13.794611 24870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:13.796277 24870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:13.796383 24870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:13.796545 24870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:13.796653 24870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:13.834321 24870 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:45031
I20260620 18:45:13.834371 24997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:45031 every 8 connection(s)
I20260620 18:45:13.835988 24870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:13.842247 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 24870
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 18:45:13.853844 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:13.838619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:60765 (local address 127.15.102.254:40393)
0620 18:45:13.838860 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:13.838868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:13.840222 (+ 1354us) server_negotiation.cc:407] Connection header received
0620 18:45:13.841982 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:13.841987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:13.842081 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:13.842191 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:13.844127 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.845311 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.847412 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.848123 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.848276 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.853491 (+ 5215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:60765: BlockingRecv error: recv got EOF from 127.15.102.195:60765 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:45:13.854759 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.839322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:60765)
0620 18:45:13.840046 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.840092 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:13.841701 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:13.842446 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:13.842455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:13.843061 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:13.843932 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.843957 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.845507 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.845511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.847269 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.847283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.848338 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.848343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.848556 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.849237 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:13.849259 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:13.853343 (+ 4084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":296,"threads_started":1}
W20260620 18:45:13.855441 24998 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:40393 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.102.254:40393: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
I20260620 18:45:13.859681 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.843380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.15.102.254:40393)
0620 18:45:13.844077 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:13.844083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:13.844101 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:45:13.844272 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:13.844276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:13.844330 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:13.844413 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:13.846208 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.847375 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.849023 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.849457 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.849591 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.852731 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:13.852743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:13.852769 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:13.852816 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:13.856589 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:13.857219 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:13.857229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:13.857238 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:13.857416 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:13.857787 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:13.857791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:13.857792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:13.858237 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:13.858531 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:13.858909 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 18:45:13.859305 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":468,"threads_started":1}
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:13.864535 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.856005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:37697)
0620 18:45:13.856200 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.856218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:13.856321 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:13.856625 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:13.856632 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:13.856958 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:13.857086 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.857094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.858253 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.858258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.859669 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.859685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.860563 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.860568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.860735 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.861386 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:13.861403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:13.864347 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 18:45:13.865024 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:13.856129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:37697 (local address 127.15.102.254:40393)
0620 18:45:13.856276 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:13.856280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:13.856295 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:45:13.856431 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:13.856435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:13.856491 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:13.856567 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:13.857239 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.858124 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.859826 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.860421 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.860526 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.864527 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:37697: BlockingRecv error: recv got EOF from 127.15.102.195:37697 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:13.872432 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.865180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:42555)
0620 18:45:13.865354 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.865371 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:13.865488 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:13.865823 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:13.865828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:13.866129 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:13.866257 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.866265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.867224 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.867267 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.868151 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.868162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.868838 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.868844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.869034 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.869510 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:13.869526 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:13.872143 (+ 2617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:45:13.872572 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:13.865242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:42555 (local address 127.15.102.254:40393)
0620 18:45:13.865593 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:13.865600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:13.865619 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:45:13.865652 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:13.865655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:13.865702 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:13.865800 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:13.866387 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.867089 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.868286 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.868700 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.868829 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.872310 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:42555: BlockingRecv error: recv got EOF from 127.15.102.195:42555 (error 108)
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":241}
W20260620 18:45:13.872766 24998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:13 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:13.971150 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:13.962768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:45065)
0620 18:45:13.963288 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:13.963310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:13.963444 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:13.963726 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:13.963729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:13.964199 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:13.964443 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.964455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.965438 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.965441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.966391 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:13.966401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.967053 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.967059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:13.967185 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.967832 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:13.967849 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:13.970808 (+ 2959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":268,"threads_started":1}
W20260620 18:45:13.971419 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:13.962900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45065 (local address 127.15.102.254:40393)
0620 18:45:13.963130 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:13.963136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:13.963395 (+ 259us) server_negotiation.cc:407] Connection header received
0620 18:45:13.963544 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:13.963547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:13.963607 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:13.963687 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:13.964593 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.965323 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.966513 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:13.966901 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:13.966992 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:13.971039 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:45065: BlockingRecv error: recv got EOF from 127.15.102.193:45065 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 18:45:14 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:14 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:14.412112 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:14.403645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:39863)
0620 18:45:14.404160 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:14.404176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:14.404286 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:14.404654 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:14.404657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:14.405099 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:14.405282 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:14.405290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.406465 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.406472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:14.407455 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:14.407466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.408216 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.408224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:14.408382 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:14.409015 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:14.409031 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:14.411817 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":287,"threads_started":1}
W20260620 18:45:14.412330 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:14.403805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:39863 (local address 127.15.102.254:40393)
0620 18:45:14.404001 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:14.404006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:14.404255 (+ 249us) server_negotiation.cc:407] Connection header received
0620 18:45:14.404435 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:14.404440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:14.404512 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:14.404602 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:14.405436 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.406314 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.407609 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.408003 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.408158 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:14.412052 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:39863: BlockingRecv error: recv got EOF from 127.15.102.194:39863 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 18:45:14 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:14 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:14.882200 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:14.873344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:42005)
0620 18:45:14.873943 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:14.873958 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:14.874125 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:14.874453 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:14.874458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:14.874938 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:14.875131 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:14.875140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.876264 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.876271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:14.877215 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:14.877226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.878227 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.878232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:14.878403 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:14.879050 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:14.879074 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:14.881847 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":359,"threads_started":1}
W20260620 18:45:14.882258 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:14.873542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:42005 (local address 127.15.102.254:40393)
0620 18:45:14.873803 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:14.873811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:14.874046 (+ 235us) server_negotiation.cc:407] Connection header received
0620 18:45:14.874262 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:14.874267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:14.874325 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:14.874429 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:14.875302 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.876127 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.877359 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.878050 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.878172 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:14.882047 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:42005: BlockingRecv error: recv got EOF from 127.15.102.195:42005 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 18:45:14 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:14 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:14.980646 25006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:14.972039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:56277)
0620 18:45:14.972527 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:14.972544 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:14.972663 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:14.972989 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:14.972992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:14.973413 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:14.973579 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:14.973587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.974770 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.974779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:14.976080 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:14.976091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.976780 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.976784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:14.976881 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:14.977494 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:14.977510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:14.980414 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":275,"threads_started":1}
W20260620 18:45:14.980922 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:14.972269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:56277 (local address 127.15.102.254:40393)
0620 18:45:14.972506 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:14.972511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:14.972622 (+ 111us) server_negotiation.cc:407] Connection header received
0620 18:45:14.972761 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:14.972765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:14.972854 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:14.972934 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:14.973732 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.974609 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.976222 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:14.976651 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:14.976755 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:14.980641 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:56277: BlockingRecv error: recv got EOF from 127.15.102.193:56277 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 18:45:15 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:15 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:15.425117 25007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:15.413139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:53267)
0620 18:45:15.413664 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:15.413680 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:15.413802 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:15.414078 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:15.414081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:15.414448 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:15.414621 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:15.414629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.415656 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.415660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:15.416546 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:15.416557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.417291 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.417298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:15.417635 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:15.421309 (+ 3674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:15.421332 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:15.424757 (+ 3425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":267,"threads_started":1}
W20260620 18:45:15.425168 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:15.413184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:53267 (local address 127.15.102.254:40393)
0620 18:45:15.413381 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:15.413390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:15.413753 (+ 363us) server_negotiation.cc:407] Connection header received
0620 18:45:15.413888 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:15.413891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:15.413952 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:15.414033 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:15.414740 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.415550 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.416668 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.417142 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.417281 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:15.424943 (+ 7662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:53267: BlockingRecv error: recv got EOF from 127.15.102.194:53267 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 18:45:15 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:15 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:15.891726 25008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:15.883183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:45447)
0620 18:45:15.883744 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:15.883760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:15.883916 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:15.884291 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:15.884295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:15.884693 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:15.884855 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:15.884863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.885849 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.885857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:15.886738 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:15.886749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.887500 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.887506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:15.887673 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:15.888336 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:15.888358 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:15.891487 (+ 3129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":349,"threads_started":1}
W20260620 18:45:15.891893 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:15.883368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:45447 (local address 127.15.102.254:40393)
0620 18:45:15.883560 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:15.883568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:15.883849 (+ 281us) server_negotiation.cc:407] Connection header received
0620 18:45:15.884051 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:15.884062 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:15.884122 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:15.884246 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:15.885007 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.885715 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.886875 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.887348 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.887476 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:15.891672 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:45447: BlockingRecv error: recv got EOF from 127.15.102.195:45447 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 18:45:15 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:15 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:15.990139 25009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:15.981648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:47335)
0620 18:45:15.982187 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:15.982205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:15.982322 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:15.982620 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:15.982623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:15.982960 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:15.983122 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:15.983130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.984215 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.984220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:15.985442 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:15.985456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.986328 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.986334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:15.986549 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:15.987143 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:15.987161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:15.989897 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":300,"threads_started":1}
W20260620 18:45:15.990342 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:15.981700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:47335 (local address 127.15.102.254:40393)
0620 18:45:15.981887 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:15.981892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:15.982284 (+ 392us) server_negotiation.cc:407] Connection header received
0620 18:45:15.982423 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:15.982427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:15.982490 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:15.982579 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:15.983269 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.984029 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.985610 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:15.986132 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:15.986271 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:15.990098 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:47335: BlockingRecv error: recv got EOF from 127.15.102.193:47335 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:45:16 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:16 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:16.434319 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:16.426122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:41303)
0620 18:45:16.426681 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:16.426701 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:16.426846 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:16.427197 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:16.427202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:16.427683 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:16.427897 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:16.427924 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.429229 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.429233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:16.430122 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:16.430133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.430727 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.430730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:16.430843 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:16.431497 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:16.431513 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:16.434078 (+ 2565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":260,"threads_started":1}
W20260620 18:45:16.434456 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:16.426228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:41303 (local address 127.15.102.254:40393)
0620 18:45:16.426449 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:16.426455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:16.426800 (+ 345us) server_negotiation.cc:407] Connection header received
0620 18:45:16.426961 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:16.426965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:16.427048 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:16.427131 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:16.428083 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.429103 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.430240 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.430613 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.430728 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:16.434244 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:41303: BlockingRecv error: recv got EOF from 127.15.102.194:41303 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:45:16 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:16 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:16.901158 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:16.892637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:33867)
0620 18:45:16.893193 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:16.893210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:16.893369 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:16.893765 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:16.893770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:16.894135 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:16.894293 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:16.894302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.895318 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.895326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:16.896287 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:16.896299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.897025 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.897031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:16.897201 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:16.897819 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:16.897837 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:16.900790 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
W20260620 18:45:16.901216 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:16.892788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:33867 (local address 127.15.102.254:40393)
0620 18:45:16.893016 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:16.893023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:16.893296 (+ 273us) server_negotiation.cc:407] Connection header received
0620 18:45:16.893557 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:16.893561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:16.893621 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:16.893729 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:16.894439 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.895135 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.896432 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.896862 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.897002 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:16.900990 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:33867: BlockingRecv error: recv got EOF from 127.15.102.195:33867 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:45:16 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:16 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:16.999907 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:16.991134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:40055)
0620 18:45:16.991627 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:16.991653 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:16.991812 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:16.992192 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:16.992196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:16.992591 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:16.992732 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:16.992740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.993747 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.993754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:16.994784 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:16.994799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.995804 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.995811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:16.996033 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:16.996733 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:16.996754 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:16.999650 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
W20260620 18:45:17.000068 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:16.991331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:40055 (local address 127.15.102.254:40393)
0620 18:45:16.991545 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:16.991551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:16.991739 (+ 188us) server_negotiation.cc:407] Connection header received
0620 18:45:16.991962 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:16.991967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:16.992039 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:16.992142 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:16.992871 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.993618 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.994940 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:16.995599 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:16.995730 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:16.999845 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:40055: BlockingRecv error: recv got EOF from 127.15.102.193:40055 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 18:45:17 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:17 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:17.443401 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:17.435431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:54783)
0620 18:45:17.435887 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:17.435903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:17.436012 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:17.436304 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:17.436307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:17.436651 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:17.436793 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:17.436802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:17.437712 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:17.437715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:17.438667 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:17.438678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:17.439447 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:17.439453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:17.439639 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:17.440198 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:17.440222 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:17.443037 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
W20260620 18:45:17.443508 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:17.435448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:54783 (local address 127.15.102.254:40393)
0620 18:45:17.435655 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:17.435664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:17.435976 (+ 312us) server_negotiation.cc:407] Connection header received
0620 18:45:17.436103 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:17.436107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:17.436182 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:17.436261 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:17.436904 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:17.437599 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:17.438789 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:17.439215 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:17.439447 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:17.443264 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:54783: BlockingRecv error: recv got EOF from 127.15.102.194:54783 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:45:17 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:17 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:17.910382 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:17.902188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:58047)
0620 18:45:17.902644 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:17.902661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:17.902782 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:17.903088 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:17.903091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:17.903453 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:17.903594 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:17.903601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:17.904810 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:17.904814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:17.905900 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:17.905913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:17.906607 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:17.906612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:17.906729 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:17.907401 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:17.907419 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:17.910153 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
W20260620 18:45:17.910600 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:17.902327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:58047 (local address 127.15.102.254:40393)
0620 18:45:17.902555 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:17.902561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:17.902739 (+ 178us) server_negotiation.cc:407] Connection header received
0620 18:45:17.902883 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:17.902887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:17.902948 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:17.903042 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:17.903756 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:17.904658 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:17.906049 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:17.906480 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:17.906584 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:17.910337 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:58047: BlockingRecv error: recv got EOF from 127.15.102.195:58047 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 18:45:18 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:18 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:18.009039 25015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:18.000899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:46723)
0620 18:45:18.001411 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:18.001428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:18.001591 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:18.001913 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:18.001918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:18.002342 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:18.002502 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:18.002510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.003596 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.003602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:18.004519 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:18.004529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.005204 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.005209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:18.005382 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:18.005972 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:18.005989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:18.008679 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":252,"threads_started":1}
W20260620 18:45:18.009110 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:18.000977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:46723 (local address 127.15.102.254:40393)
0620 18:45:18.001207 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:18.001215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:18.001528 (+ 313us) server_negotiation.cc:407] Connection header received
0620 18:45:18.001718 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:18.001730 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:18.001788 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:18.001888 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:18.002651 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.003471 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.004664 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.005065 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.005196 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:18.008866 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:46723: BlockingRecv error: recv got EOF from 127.15.102.193:46723 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 18:45:18 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:18 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:18.453259 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:18.444419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:60349)
0620 18:45:18.444967 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:18.444983 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:18.445136 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:18.445490 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:18.445495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:18.445897 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:18.446045 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:18.446053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.447094 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.447101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:18.448086 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:18.448097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.448813 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.448819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:18.448980 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:18.449651 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:18.449710 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:18.452914 (+ 3204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":277,"threads_started":1}
W20260620 18:45:18.454214 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:18.444655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:60349 (local address 127.15.102.254:40393)
0620 18:45:18.444843 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:18.444850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:18.445070 (+ 220us) server_negotiation.cc:407] Connection header received
0620 18:45:18.445256 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:18.445261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:18.445334 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:18.445437 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:18.446198 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.446952 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.448226 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.448654 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.448790 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:18.453317 (+ 4527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:60349: BlockingRecv error: recv got EOF from 127.15.102.194:60349 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 18:45:18 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:18 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:18.919979 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:18.911473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:45483)
0620 18:45:18.912016 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:18.912034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:18.912141 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:18.912486 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:18.912489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:18.912872 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:18.913020 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:18.913028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.914456 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.914462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:18.915495 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:18.915507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.916134 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.916138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:18.916251 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:18.916886 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:18.916903 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:18.919730 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":281,"threads_started":1}
W20260620 18:45:18.920238 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:18.911567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:45483 (local address 127.15.102.254:40393)
0620 18:45:18.911738 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:18.911742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:18.912114 (+ 372us) server_negotiation.cc:407] Connection header received
0620 18:45:18.912266 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:18.912269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:18.912327 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:18.912454 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:18.913138 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.914275 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.915615 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:18.916017 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:18.916110 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:18.919945 (+ 3835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:45483: BlockingRecv error: recv got EOF from 127.15.102.195:45483 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:45:19 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:19 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:19.018618 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:19.010241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:47305)
0620 18:45:19.010823 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:19.010844 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:19.011008 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:19.011396 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:19.011403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:19.011787 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:19.011942 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:19.011950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.013020 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.013023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:19.014051 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:19.014063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.014808 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.014813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:19.014943 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:19.015662 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:19.015679 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:19.018394 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":306,"threads_started":1}
W20260620 18:45:19.019002 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:19.010301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:47305 (local address 127.15.102.254:40393)
0620 18:45:19.010551 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:19.010560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:19.010943 (+ 383us) server_negotiation.cc:407] Connection header received
0620 18:45:19.011137 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:19.011142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:19.011250 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:19.011367 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:19.012063 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.012898 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.014177 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.014675 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.014769 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:19.018554 (+ 3785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:47305: BlockingRecv error: recv got EOF from 127.15.102.193:47305 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 18:45:19 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:19 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:19.465515 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:19.454393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:57985)
0620 18:45:19.455092 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:19.455113 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:19.455632 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:19.456192 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:19.456198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:19.456999 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:19.457213 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:19.457227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.458183 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.458190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:19.459901 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:19.459917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.460662 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.460670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:19.460894 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:19.461648 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:19.461845 (+ 197us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:19.465298 (+ 3453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":430,"threads_started":1}
W20260620 18:45:19.465776 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:19.454458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:57985 (local address 127.15.102.254:40393)
0620 18:45:19.454647 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:19.454655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:19.455218 (+ 563us) server_negotiation.cc:407] Connection header received
0620 18:45:19.455796 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:19.455805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:19.455891 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:19.455973 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:19.457344 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.458043 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.460041 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.460511 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.460602 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:19.465540 (+ 4938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:57985: BlockingRecv error: recv got EOF from 127.15.102.194:57985 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:45:19 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:19 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:19.929278 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:19.920806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:43579)
0620 18:45:19.921301 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:19.921318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:19.921442 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:19.921733 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:19.921736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:19.922099 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:19.922247 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:19.922255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.923251 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.923256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:19.924686 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:19.924700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.925492 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.925498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:19.925665 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:19.926284 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:19.926306 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:19.929065 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":290,"threads_started":1}
W20260620 18:45:19.929461 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:19.920907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:43579 (local address 127.15.102.254:40393)
0620 18:45:19.921142 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:19.921148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:19.921398 (+ 250us) server_negotiation.cc:407] Connection header received
0620 18:45:19.921545 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:19.921548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:19.921607 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:19.921684 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:19.922370 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.923091 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.924865 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:19.925337 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:19.925436 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:19.929236 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:43579: BlockingRecv error: recv got EOF from 127.15.102.195:43579 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 18:45:20 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:20 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:20.028072 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:20.019566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:35783)
0620 18:45:20.020121 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:20.020138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:20.020290 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:20.020629 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:20.020634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:20.021059 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:20.021216 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:20.021224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.022227 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.022234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:20.023158 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:20.023168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.023924 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.023930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:20.024101 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:20.024663 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:20.024680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:20.027695 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":254,"threads_started":1}
W20260620 18:45:20.028153 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:20.019615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:35783 (local address 127.15.102.254:40393)
0620 18:45:20.019846 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:20.019855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:20.020227 (+ 372us) server_negotiation.cc:407] Connection header received
0620 18:45:20.020421 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:20.020425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:20.020482 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:20.020590 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:20.021365 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.022092 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.023366 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.023769 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.023901 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:20.027902 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:35783: BlockingRecv error: recv got EOF from 127.15.102.193:35783 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 18:45:20 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:20 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:20.480175 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:20.466563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:35785)
0620 18:45:20.467163 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:20.467180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:20.467383 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:20.467724 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:20.467727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:20.468134 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:20.468316 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:20.468326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.469314 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.469318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:20.470296 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:20.470308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.470968 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.470972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:20.471087 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:20.476347 (+ 5260us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:20.476366 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:20.479771 (+ 3405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":293,"threads_started":1}
W20260620 18:45:20.480273 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:20.466601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:35785 (local address 127.15.102.254:40393)
0620 18:45:20.466789 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:20.466797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:20.467283 (+ 486us) server_negotiation.cc:407] Connection header received
0620 18:45:20.467513 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:20.467517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:20.467575 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:20.467654 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:20.468464 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.469183 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.470458 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.470834 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.470927 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:20.479982 (+ 9055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:35785: BlockingRecv error: recv got EOF from 127.15.102.194:35785 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 18:45:20 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:20 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:20.938175 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:20.930167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:59295)
0620 18:45:20.930675 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:20.930691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:20.930854 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:20.931177 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:20.931181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:20.931685 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:20.931844 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:20.931853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.932851 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.932859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:20.933767 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:20.933778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.934480 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.934485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:20.934645 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:20.935263 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:20.935281 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:20.937917 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":241,"threads_started":1}
W20260620 18:45:20.938289 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:20.930222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:59295 (local address 127.15.102.254:40393)
0620 18:45:20.930432 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:20.930440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:20.930772 (+ 332us) server_negotiation.cc:407] Connection header received
0620 18:45:20.930981 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:20.930986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:20.931054 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:20.931153 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:20.931986 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.932709 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.933903 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:20.934325 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:20.934455 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:20.938105 (+ 3650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:59295: BlockingRecv error: recv got EOF from 127.15.102.195:59295 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 18:45:21 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:21 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:21.037473 25024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:21.029066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:53571)
0620 18:45:21.029570 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:21.029587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:21.029696 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:21.029969 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:21.029973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:21.030339 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:21.030502 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:21.030510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.031509 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.031513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:21.032451 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:21.032462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.033167 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.033171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:21.033278 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:21.033889 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:21.033905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:21.037116 (+ 3211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":277,"threads_started":1}
W20260620 18:45:21.037583 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:21.029260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:53571 (local address 127.15.102.254:40393)
0620 18:45:21.029512 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:21.029520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:21.029640 (+ 120us) server_negotiation.cc:407] Connection header received
0620 18:45:21.029776 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:21.029780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:21.029836 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:21.029935 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:21.030629 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.031360 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.032594 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.033015 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.033108 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:21.037324 (+ 4216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:53571: BlockingRecv error: recv got EOF from 127.15.102.193:53571 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 18:45:21 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:21 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:21.489837 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:21.481298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:50449)
0620 18:45:21.481774 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:21.481791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:21.481951 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:21.482290 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:21.482295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:21.482714 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:21.482869 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:21.482879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.484133 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.484139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:21.485075 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:21.485087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.485822 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.485828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:21.485989 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:21.486627 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:21.486646 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:21.489493 (+ 2847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":255,"threads_started":1}
W20260620 18:45:21.489916 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:21.481363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:50449 (local address 127.15.102.254:40393)
0620 18:45:21.481589 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:21.481596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:21.481901 (+ 305us) server_negotiation.cc:407] Connection header received
0620 18:45:21.482084 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:21.482088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:21.482147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:21.482256 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:21.483013 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.483991 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.485217 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.485636 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.485777 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:21.489697 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:50449: BlockingRecv error: recv got EOF from 127.15.102.194:50449 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 18:45:21 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:21 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:21.947521 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:21.939288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:48767)
0620 18:45:21.939825 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:21.939840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:21.939996 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:21.940431 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:21.940436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:21.940969 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:21.941164 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:21.941176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.942267 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.942271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:21.943177 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:21.943187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.943854 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.943858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:21.943961 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:21.944540 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:21.944556 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:21.947211 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":296,"threads_started":1}
W20260620 18:45:21.947682 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:21.939421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:48767 (local address 127.15.102.254:40393)
0620 18:45:21.939665 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:21.939672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:21.939930 (+ 258us) server_negotiation.cc:407] Connection header received
0620 18:45:21.940140 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:21.940144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:21.940246 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:21.940349 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:21.941328 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.942133 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.943313 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:21.943707 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:21.943798 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:21.947444 (+ 3646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:48767: BlockingRecv error: recv got EOF from 127.15.102.195:48767 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 18:45:22 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:22 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:22.046957 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:22.038496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:54443)
0620 18:45:22.039341 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:22.039357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:22.039514 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:22.039851 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:22.039856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:22.040241 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:22.040397 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:22.040405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.041401 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.041409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:22.042330 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:22.042340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.043069 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.043074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:22.043273 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:22.043853 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:22.043869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:22.046612 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":577,"threads_started":1}
W20260620 18:45:22.047080 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:22.038602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:54443 (local address 127.15.102.254:40393)
0620 18:45:22.038880 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:22.038889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:22.039445 (+ 556us) server_negotiation.cc:407] Connection header received
0620 18:45:22.039637 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:22.039643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:22.039710 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:22.039809 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:22.040542 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.041259 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.042467 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.042913 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.043039 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:22.046828 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:54443: BlockingRecv error: recv got EOF from 127.15.102.193:54443 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 18:45:22 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:22 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:22.499727 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:22.490995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:45371)
0620 18:45:22.491585 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:22.491605 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:22.491753 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:22.492120 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:22.492127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:22.492663 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:22.492888 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:22.492900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.493963 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.493968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:22.495084 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:22.495097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.495822 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.495827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:22.495958 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:22.496568 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:22.496586 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:22.499472 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":296,"threads_started":1}
W20260620 18:45:22.499920 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:22.491032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:45371 (local address 127.15.102.254:40393)
0620 18:45:22.491286 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:22.491292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:22.491714 (+ 422us) server_negotiation.cc:407] Connection header received
0620 18:45:22.491868 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:22.491873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:22.491955 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:22.492056 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:22.493031 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.493814 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.495249 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.495680 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.495782 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:22.499690 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:45371: BlockingRecv error: recv got EOF from 127.15.102.194:45371 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 18:45:22 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:22 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:22.957171 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:22.948445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:52447)
0620 18:45:22.949012 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:22.949031 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:22.949150 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:22.949482 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:22.949486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:22.949874 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:22.950045 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:22.950054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.951430 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.951434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:22.952544 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:22.952556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.953335 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.953340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:22.953473 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:22.954036 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:22.954055 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:22.956928 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
W20260620 18:45:22.957315 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:22.948533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:52447 (local address 127.15.102.254:40393)
0620 18:45:22.948696 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:22.948700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:22.949106 (+ 406us) server_negotiation.cc:407] Connection header received
0620 18:45:22.949255 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:22.949259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:22.949318 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:22.949423 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:22.950215 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.951285 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.952699 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:22.953164 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:22.953279 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:22.957106 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:52447: BlockingRecv error: recv got EOF from 127.15.102.195:52447 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 18:45:23 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:23 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:23.056929 25030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:23.048377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:54241)
0620 18:45:23.048879 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:23.048895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:23.049057 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:23.049449 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:23.049454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:23.050144 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:23.050299 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:23.050308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.051351 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.051354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:23.052235 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:23.052245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.052963 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.052968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:23.053103 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:23.053747 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:23.053767 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:23.056703 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
W20260620 18:45:23.057183 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:23.048466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:54241 (local address 127.15.102.254:40393)
0620 18:45:23.048667 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:23.048676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:23.048996 (+ 320us) server_negotiation.cc:407] Connection header received
0620 18:45:23.049180 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:23.049186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:23.049265 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:23.049376 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:23.050434 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.051161 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.052380 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.052795 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.052887 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:23.056895 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:54241: BlockingRecv error: recv got EOF from 127.15.102.193:54241 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 18:45:23 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:23 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:23.508599 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:23.500759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:40325)
0620 18:45:23.501304 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:23.501321 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:23.501434 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:23.501721 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:23.501724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:23.502078 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:23.502225 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:23.502233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.503178 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.503182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:23.504132 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:23.504143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.504800 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.504804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:23.504924 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:23.505587 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:23.505604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:23.508374 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":323,"threads_started":1}
W20260620 18:45:23.508796 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:23.500893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:40325 (local address 127.15.102.254:40393)
0620 18:45:23.501073 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:23.501077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:23.501399 (+ 322us) server_negotiation.cc:407] Connection header received
0620 18:45:23.501528 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:23.501532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:23.501592 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:23.501678 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:23.502359 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.503046 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.504248 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.504666 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.504758 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:23.508565 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:40325: BlockingRecv error: recv got EOF from 127.15.102.194:40325 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:45:23 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:23 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:23.966646 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:23.958180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:50007)
0620 18:45:23.958739 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:23.958755 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:23.958862 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:23.959208 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:23.959212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:23.959594 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:23.959736 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:23.959745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.960778 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.960782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:23.961817 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:23.961829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.962487 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.962491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:23.962593 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:23.963278 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:23.963310 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:23.966308 (+ 2998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":323,"threads_started":1}
W20260620 18:45:23.966926 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:23.958470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:50007 (local address 127.15.102.254:40393)
0620 18:45:23.958719 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:23.958725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:23.958837 (+ 112us) server_negotiation.cc:407] Connection header received
0620 18:45:23.958975 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:23.958979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:23.959037 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:23.959120 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:23.959891 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.960650 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.961949 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:23.962358 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:23.962452 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:23.966495 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:50007: BlockingRecv error: recv got EOF from 127.15.102.195:50007 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Jun 20 18:45:24 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:24 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:24.066324 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:24.057892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:51799)
0620 18:45:24.058439 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:24.058456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:24.058564 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:24.058856 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:24.058860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:24.059318 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:24.059488 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:24.059497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.060528 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.060532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:24.061424 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:24.061434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.062086 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.062090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:24.062211 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:24.062810 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:24.062827 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:24.065977 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":297,"threads_started":1}
W20260620 18:45:24.066368 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:24.058014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:51799 (local address 127.15.102.254:40393)
0620 18:45:24.058171 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:24.058175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:24.058523 (+ 348us) server_negotiation.cc:407] Connection header received
0620 18:45:24.058655 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:24.058659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:24.058732 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:24.058815 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:24.059600 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.060377 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.061550 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.061949 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.062037 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:24.066159 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:51799: BlockingRecv error: recv got EOF from 127.15.102.193:51799 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 18:45:24 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:24 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:24.518934 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:24.509764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:37085)
0620 18:45:24.510308 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:24.510326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:24.510435 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:24.510713 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:24.510717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:24.511139 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:24.511438 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:24.511452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.512669 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.512673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:24.513791 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:24.513803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.514498 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.514510 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:24.514628 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:24.515315 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:24.515342 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:24.518569 (+ 3227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
W20260620 18:45:24.519019 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:24.509867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:37085 (local address 127.15.102.254:40393)
0620 18:45:24.510038 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:24.510042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:24.510400 (+ 358us) server_negotiation.cc:407] Connection header received
0620 18:45:24.510531 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:24.510535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:24.510599 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:24.510679 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:24.511595 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.512551 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.513930 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.514347 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.514449 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:24.518752 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:37085: BlockingRecv error: recv got EOF from 127.15.102.194:37085 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 18:45:24 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:24 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:24.975566 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:24.967568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:60383)
0620 18:45:24.968018 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:24.968034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:24.968149 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:24.968435 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:24.968439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:24.968810 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:24.968974 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:24.968982 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.969974 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.969978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:24.970885 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:24.970896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.971597 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.971601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:24.971720 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:24.972348 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:24.972366 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:24.975294 (+ 2928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
W20260620 18:45:24.975697 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:24.967610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:60383 (local address 127.15.102.254:40393)
0620 18:45:24.967835 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:24.967845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:24.968125 (+ 280us) server_negotiation.cc:407] Connection header received
0620 18:45:24.968246 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:24.968250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:24.968311 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:24.968400 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:24.969109 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.969839 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.971006 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:24.971468 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:24.971561 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:24.975464 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:60383: BlockingRecv error: recv got EOF from 127.15.102.195:60383 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:45:25 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:25 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:25.075815 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:25.067230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:42377)
0620 18:45:25.067742 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:25.067766 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:25.067927 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:25.068268 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:25.068273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:25.068680 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:25.068829 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:25.068837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.069837 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.069842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:25.070740 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:25.070750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.071582 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.071588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:25.071773 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.072395 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:25.072413 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:25.075580 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":277,"threads_started":1}
W20260620 18:45:25.076046 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:25.067340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:42377 (local address 127.15.102.254:40393)
0620 18:45:25.067568 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:25.067577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:25.067847 (+ 270us) server_negotiation.cc:407] Connection header received
0620 18:45:25.068076 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:25.068080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:25.068137 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:25.068236 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:25.068962 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.069693 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.070893 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.071435 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.071561 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.075776 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:42377: BlockingRecv error: recv got EOF from 127.15.102.193:42377 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 18:45:25 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:25 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:25.530972 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:25.519854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:45117)
0620 18:45:25.520357 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:25.520373 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:25.520482 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:25.520761 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:25.520764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:25.521147 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:25.521292 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:25.521299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.522376 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.522380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:25.523452 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:25.523463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.524093 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.524097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:25.524213 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.527828 (+ 3615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:25.527851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:25.530737 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":291,"threads_started":1}
W20260620 18:45:25.531132 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:25.519884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:45117 (local address 127.15.102.254:40393)
0620 18:45:25.520069 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:25.520073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:25.520445 (+ 372us) server_negotiation.cc:407] Connection header received
0620 18:45:25.520575 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:25.520579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:25.520639 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:25.520720 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:25.521425 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.522220 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.523570 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.523977 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.524071 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.530933 (+ 6862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:45117: BlockingRecv error: recv got EOF from 127.15.102.194:45117 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 18:45:25 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:25 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:25.984936 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:25.976645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:57445)
0620 18:45:25.977173 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:25.977194 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:25.977379 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:25.977777 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:25.977780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:25.978233 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:25.978384 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:25.978392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.979450 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.979454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:25.980351 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:25.980361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.981076 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.981081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:25.981234 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.981866 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:25.981884 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:25.984692 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":299,"threads_started":1}
W20260620 18:45:25.985138 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:25.976722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:57445 (local address 127.15.102.254:40393)
0620 18:45:25.976932 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:25.976940 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:25.977313 (+ 373us) server_negotiation.cc:407] Connection header received
0620 18:45:25.977540 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:25.977544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:25.977608 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:25.977723 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:25.978517 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.979287 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.980497 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:25.980921 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:25.981019 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:25.984899 (+ 3880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:57445: BlockingRecv error: recv got EOF from 127.15.102.195:57445 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 18:45:26 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:26 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:26.084690 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:26.076701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:34911)
0620 18:45:26.077256 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:26.077272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:26.077390 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:26.077686 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:26.077689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:26.078048 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:26.078195 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:26.078203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.079153 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.079157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:26.080127 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:26.080138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.080876 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.080880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:26.081000 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:26.081614 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:26.081632 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:26.084446 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":302,"threads_started":1}
W20260620 18:45:26.084920 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:26.076821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:34911 (local address 127.15.102.254:40393)
0620 18:45:26.077004 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:26.077008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:26.077347 (+ 339us) server_negotiation.cc:407] Connection header received
0620 18:45:26.077496 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:26.077499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:26.077558 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:26.077643 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:26.078317 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.079036 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.080250 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.080691 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.080786 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:26.084645 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:34911: BlockingRecv error: recv got EOF from 127.15.102.193:34911 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 18:45:26 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:26 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:26.540316 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:26.532076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:46053)
0620 18:45:26.532634 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:26.532649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:26.532815 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:26.533148 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:26.533152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:26.533563 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:26.533711 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:26.533720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.534953 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.534959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:26.535901 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:26.535911 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.536640 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.536646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:26.536810 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:26.537361 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:26.537380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:26.540051 (+ 2671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":308,"threads_started":1}
W20260620 18:45:26.540474 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:26.532301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:46053 (local address 127.15.102.254:40393)
0620 18:45:26.532504 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:26.532511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:26.532743 (+ 232us) server_negotiation.cc:407] Connection header received
0620 18:45:26.532941 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:26.532947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:26.533020 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:26.533121 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:26.533875 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.534841 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.536044 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.536467 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.536604 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:26.540235 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:46053: BlockingRecv error: recv got EOF from 127.15.102.194:46053 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 18:45:26 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:26 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:26.993680 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:26.985782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:59325)
0620 18:45:26.986297 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:26.986313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:26.986420 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:26.986747 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:26.986751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:26.987116 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:26.987329 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:26.987338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.988303 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.988307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:26.989215 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:26.989225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.989970 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.989975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:26.990092 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:26.990697 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:26.990718 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:26.993427 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":271,"threads_started":1}
W20260620 18:45:26.993770 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:26.985872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:59325 (local address 127.15.102.254:40393)
0620 18:45:26.986034 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:26.986038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:26.986383 (+ 345us) server_negotiation.cc:407] Connection header received
0620 18:45:26.986516 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:26.986520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:26.986578 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:26.986677 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:26.987475 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.988177 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.989340 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:26.989792 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:26.989886 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:26.993609 (+ 3723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:59325: BlockingRecv error: recv got EOF from 127.15.102.195:59325 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 18:45:27 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:27 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:27.093954 25042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:27.085602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:45257)
0620 18:45:27.086053 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:27.086071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:27.086189 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:27.086507 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:27.086510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:27.086866 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:27.087033 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:27.087040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.088196 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.088199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:27.089102 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:27.089113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.089800 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.089804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:27.089915 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:27.090574 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:27.090591 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:27.093583 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":261,"threads_started":1}
W20260620 18:45:27.094120 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:27.085681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45257 (local address 127.15.102.254:40393)
0620 18:45:27.086050 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:27.086060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:27.086179 (+ 119us) server_negotiation.cc:407] Connection header received
0620 18:45:27.086293 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:27.086297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:27.086380 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:27.086473 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:27.087206 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.088056 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.089254 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.089674 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.089778 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:27.093815 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:45257: BlockingRecv error: recv got EOF from 127.15.102.193:45257 (error 108)
Metrics: {"server-negotiator.queue_time_us":240}
Jun 20 18:45:27 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:27 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:27.550226 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:27.541427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:50919)
0620 18:45:27.541954 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:27.541978 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:27.542196 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:27.542584 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:27.542591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:27.543169 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:27.543480 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:27.543498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.544529 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.544533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:27.545493 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:27.545505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.546180 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.546184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:27.546305 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:27.547026 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:27.547046 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:27.549863 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":248,"threads_started":1}
W20260620 18:45:27.550264 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:27.541536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:50919 (local address 127.15.102.254:40393)
0620 18:45:27.541751 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:27.541759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:27.542085 (+ 326us) server_negotiation.cc:407] Connection header received
0620 18:45:27.542346 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:27.542351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:27.542428 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:27.542549 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:27.543657 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.544387 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.545625 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.546032 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.546120 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:27.550049 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:50919: BlockingRecv error: recv got EOF from 127.15.102.194:50919 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 18:45:27 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:27 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.195@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:28.002142 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:27.994522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.195:49311)
0620 18:45:27.995015 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:27.995031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:27.995142 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:27.995493 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:27.995496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:27.995843 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:27.996013 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:27.996021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.996946 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.996950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:27.997823 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:27.997832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.998462 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.998466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:27.998578 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:27.999163 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:27.999179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:28.001887 (+ 2708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":272,"threads_started":1}
W20260620 18:45:28.002312 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:27.994664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:49311 (local address 127.15.102.254:40393)
0620 18:45:27.994854 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:27.994858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:27.995106 (+ 248us) server_negotiation.cc:407] Connection header received
0620 18:45:27.995285 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:27.995292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:27.995363 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:27.995448 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:27.996141 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.996829 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.997936 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:27.998346 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:27.998434 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.002074 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.195:49311: BlockingRecv error: recv got EOF from 127.15.102.195:49311 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 18:45:28 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:28 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.193@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:28.103456 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.095340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.193:50877)
0620 18:45:28.095904 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:28.095921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:28.096026 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:28.096342 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:28.096345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:28.096706 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:28.096861 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:28.096869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.097821 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.097825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:28.098714 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:28.098724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.099537 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.099541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:28.099716 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.100426 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:28.100447 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:28.103139 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":290,"threads_started":1}
W20260620 18:45:28.103546 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:28.095443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:50877 (local address 127.15.102.254:40393)
0620 18:45:28.095661 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:28.095666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:28.095998 (+ 332us) server_negotiation.cc:407] Connection header received
0620 18:45:28.096121 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:28.096124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:28.096196 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:28.096295 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:28.096979 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.097697 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.098833 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.099387 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.099518 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.103374 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.193:50877: BlockingRecv error: recv got EOF from 127.15.102.193:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 20 18:45:28 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 18:45:28 dist-test-slave-zfp8 krb5kdc[24000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.102.194@KRBTEST.COM for oryx/127.15.102.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 18:45:28.559542 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.551524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40393 (local address 127.15.102.194:50905)
0620 18:45:28.552008 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:28.552025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:28.552140 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:28.552442 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:28.552445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:28.552818 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:28.552982 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:28.552990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.553925 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.553928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:28.554838 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:28.554849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.555522 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.555527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:28.555644 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.556256 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:28.556272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:28.559296 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.102.254:40393: Server oryx/127.15.102.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":267,"threads_started":1}
W20260620 18:45:28.559711 24737 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:28.551604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:50905 (local address 127.15.102.254:40393)
0620 18:45:28.551859 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:28.551865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:28.552104 (+ 239us) server_negotiation.cc:407] Connection header received
0620 18:45:28.552249 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:28.552253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:28.552311 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:28.552394 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:28.553098 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.553801 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.554952 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.555406 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.555494 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.559490 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.102.194:50905: BlockingRecv error: recv got EOF from 127.15.102.194:50905 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 18:45:28.843568 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24588
I20260620 18:45:28.855852 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.844708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.15.102.193:38639)
0620 18:45:28.845264 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:28.845271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:28.845305 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:45:28.845365 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:28.845372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:28.845604 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:28.845768 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:28.847173 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.848312 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.849586 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.850125 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.850222 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.852770 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:28.852778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:28.852792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:28.852827 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:28.855363 (+ 2536us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.102.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:45:28.855622 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41454: Request ticket server kudu/127.15.102.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
W20260620 18:45:28.855978 25048 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41454: Request ticket server kudu/127.15.102.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:45:28.855988 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24588: Not authorized: Client connection negotiation failed: client connection to 127.15.102.193:38639: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.102.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:45:28.856178 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 24588
I20260620 18:45:28.873039 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24739
I20260620 18:45:28.885514 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.874271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.15.102.194:34951)
0620 18:45:28.874927 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:28.874935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:28.874966 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:45:28.875026 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:28.875032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:28.875389 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:28.875560 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:28.876769 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.877846 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.879148 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.879786 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.879889 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.882323 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:28.882332 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:28.882347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:28.882380 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:28.884964 (+ 2584us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.102.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:45:28.885230 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33930: Request ticket server kudu/127.15.102.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":353,"threads_started":1}
W20260620 18:45:28.885620 25049 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33930: Request ticket server kudu/127.15.102.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:45:28.885727 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24739: Not authorized: Client connection negotiation failed: client connection to 127.15.102.194:34951: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.102.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:45:28.885886 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 24739
I20260620 18:45:28.902536 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24870
I20260620 18:45:28.914655 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:28.903561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.15.102.195:45031)
0620 18:45:28.904145 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:28.904152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:28.904180 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:45:28.904241 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:28.904248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:28.904482 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:28.904640 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:28.905877 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.906963 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.908258 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:28.908817 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:28.908918 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:28.911480 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:28.911489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:28.911523 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:28.911564 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:28.914091 (+ 2527us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.102.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 18:45:28.914416 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53672: Request ticket server kudu/127.15.102.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":294,"threads_started":1}
W20260620 18:45:28.914773 25050 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53672: Request ticket server kudu/127.15.102.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 18:45:28.914762 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24870: Not authorized: Client connection negotiation failed: client connection to 127.15.102.195:45031: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.102.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 18:45:28.914911 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 24870
I20260620 18:45:28.930965 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 24519
I20260620 18:45:28.932283 24581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:29.045990 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 24519
2026-06-20T18:45:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21030 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T18:45:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:29Z Disabled control of system clock
I20260620 18:45:29.121723 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35123
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:35123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 18:45:29.441532 25055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:29.441809 25055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:29.441941 25055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:29.448971 25055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:29.449077 25055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:29.449150 25055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:29.449241 25055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:29.449323 25055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:29.459067 25055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:35123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:29.460495 25055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:29.462270 25055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:29.475704 25060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:29.475797 25063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:29.475777 25061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:29.477972 25055 server_base.cc:1034] running on GCE node
I20260620 18:45:29.478792 25055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:29.480350 25055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:29.481726 25055 hybrid_clock.cc:648] HybridClock initialized: now 1781981129481682 us; error 38 us; skew 500 ppm
I20260620 18:45:29.482101 25055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:45:29.484477 25055 webserver.cc:460] Webserver started at http://127.15.102.254:34095/ using document root <none> and password file <none>
I20260620 18:45:29.485200 25055 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:29.485330 25055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:29.485657 25055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:29.488691 25055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "af8266329d6c49d1a67410a31b971bc9"
format_stamp: "Formatted at 2026-06-20 18:45:29 on dist-test-slave-zfp8"
server_key: "dba71a44c3d133756ee16e98f19554e9"
server_key_iv: "56cfc97d4df80126d87b49a8ae23185f"
server_key_version: "encryptionkey@0"
I20260620 18:45:29.489470 25055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "af8266329d6c49d1a67410a31b971bc9"
format_stamp: "Formatted at 2026-06-20 18:45:29 on dist-test-slave-zfp8"
server_key: "dba71a44c3d133756ee16e98f19554e9"
server_key_iv: "56cfc97d4df80126d87b49a8ae23185f"
server_key_version: "encryptionkey@0"
I20260620 18:45:29.494560 25055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 18:45:29.498072 25069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:29.499583 25055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:29.499786 25055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "af8266329d6c49d1a67410a31b971bc9"
format_stamp: "Formatted at 2026-06-20 18:45:29 on dist-test-slave-zfp8"
server_key: "dba71a44c3d133756ee16e98f19554e9"
server_key_iv: "56cfc97d4df80126d87b49a8ae23185f"
server_key_version: "encryptionkey@0"
I20260620 18:45:29.500021 25055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:29.515812 25055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:29.516641 25055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:29.516925 25055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:29.535395 25055 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:35123
I20260620 18:45:29.535429 25120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:35123 every 8 connection(s)
I20260620 18:45:29.536731 25055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:29.540376 25121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:29.541860 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 25055
I20260620 18:45:29.542057 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:45:29.542687 15771 external_mini_cluster.cc:1428] Setting key f18d306ee9fb195f44cb44b2dbbf7ec3
I20260620 18:45:29.551656 25121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9: Bootstrap starting.
I20260620 18:45:29.555258 25121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:29.556321 25121 log.cc:824] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:29.556622 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:29.544752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.15.102.254:35123)
0620 18:45:29.545699 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:29.545709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:29.545751 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:45:29.547442 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:29.547464 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:29.547898 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:29.548281 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:29.548851 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.550145 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.552077 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:29.552733 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:29.552861 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:29.554171 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:29.554178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:29.554191 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:29.554719 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:29.554804 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:29.554823 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:29.554912 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:29.555168 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 18:45:29.555609 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":322,"threads_started":1}
I20260620 18:45:29.559186 25121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9: No bootstrap required, opened a new log
I20260620 18:45:29.566035 25121 raft_consensus.cc:348] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8266329d6c49d1a67410a31b971bc9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35123 } }
I20260620 18:45:29.566406 25121 raft_consensus.cc:374] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:29.566514 25121 raft_consensus.cc:729] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8266329d6c49d1a67410a31b971bc9, State: Initialized, Role: FOLLOWER
I20260620 18:45:29.567055 25121 consensus_queue.cc:260] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8266329d6c49d1a67410a31b971bc9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35123 } }
I20260620 18:45:29.567304 25121 raft_consensus.cc:388] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:29.567461 25121 raft_consensus.cc:482] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:29.567631 25121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:29.569669 25121 raft_consensus.cc:504] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8266329d6c49d1a67410a31b971bc9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35123 } }
I20260620 18:45:29.570184 25121 leader_election.cc:302] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af8266329d6c49d1a67410a31b971bc9; no voters:
I20260620 18:45:29.570742 25121 leader_election.cc:288] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:45:29.570919 25126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:29.571669 25126 raft_consensus.cc:686] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [term 1 LEADER]: Becoming Leader. State: Replica: af8266329d6c49d1a67410a31b971bc9, State: Running, Role: LEADER
I20260620 18:45:29.572203 25126 consensus_queue.cc:237] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8266329d6c49d1a67410a31b971bc9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35123 } }
I20260620 18:45:29.572930 25121 sys_catalog.cc:564] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:29.574913 25128 sys_catalog.cc:455] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af8266329d6c49d1a67410a31b971bc9. Latest consensus state: current_term: 1 leader_uuid: "af8266329d6c49d1a67410a31b971bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8266329d6c49d1a67410a31b971bc9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35123 } } }
I20260620 18:45:29.575302 25128 sys_catalog.cc:458] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:29.579344 25127 sys_catalog.cc:455] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af8266329d6c49d1a67410a31b971bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8266329d6c49d1a67410a31b971bc9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35123 } } }
I20260620 18:45:29.579733 25127 sys_catalog.cc:458] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:29.583263 25131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:29.588786 25131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:29.597656 25131 catalog_manager.cc:1347] Generated new cluster ID: edfa5ff557dc4db5b4af60305b59dc8c
I20260620 18:45:29.597791 25131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:29.606526 25131 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:45:29.607853 25131 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:29.616587 25131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af8266329d6c49d1a67410a31b971bc9: Generated new TSK 0
I20260620 18:45:29.617612 25131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:45:29.631587 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35123
--builtin_ntp_servers=127.15.102.212:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 18:45:29.955476 25145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:29.955745 25145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:29.955904 25145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:29.962963 25145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:29.963054 25145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:29.963227 25145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:45:29.972064 25145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:29.973461 25145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:29.975358 25145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:29.988452 25150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:29.988533 25153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:29.988502 25151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:29.989508 25145 server_base.cc:1034] running on GCE node
I20260620 18:45:29.990181 25145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:29.991088 25145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:29.992306 25145 hybrid_clock.cc:648] HybridClock initialized: now 1781981129992242 us; error 59 us; skew 500 ppm
I20260620 18:45:29.992619 25145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:45:29.994809 25145 webserver.cc:460] Webserver started at http://127.15.102.193:33683/ using document root <none> and password file <none>
I20260620 18:45:29.995579 25145 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:29.995710 25145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:29.996032 25145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:29.998919 25145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "e1d3b9f3fd3b426ebf1712c6e61b2688"
format_stamp: "Formatted at 2026-06-20 18:45:29 on dist-test-slave-zfp8"
server_key: "be38cc466f0854cb86146ad699a0e6b9"
server_key_iv: "0d48bd74d8e674af9de2850c72474034"
server_key_version: "encryptionkey@0"
I20260620 18:45:29.999737 25145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "e1d3b9f3fd3b426ebf1712c6e61b2688"
format_stamp: "Formatted at 2026-06-20 18:45:29 on dist-test-slave-zfp8"
server_key: "be38cc466f0854cb86146ad699a0e6b9"
server_key_iv: "0d48bd74d8e674af9de2850c72474034"
server_key_version: "encryptionkey@0"
I20260620 18:45:30.004644 25145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:45:30.008116 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.009459 25145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:30.009629 25145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "e1d3b9f3fd3b426ebf1712c6e61b2688"
format_stamp: "Formatted at 2026-06-20 18:45:29 on dist-test-slave-zfp8"
server_key: "be38cc466f0854cb86146ad699a0e6b9"
server_key_iv: "0d48bd74d8e674af9de2850c72474034"
server_key_version: "encryptionkey@0"
I20260620 18:45:30.009855 25145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:30.029668 25145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:30.030531 25145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:30.030840 25145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:30.031889 25145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:30.033314 25145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:30.033442 25145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.033573 25145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:30.033670 25145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.066232 25145 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:33115
I20260620 18:45:30.066282 25271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:33115 every 8 connection(s)
I20260620 18:45:30.067668 25145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:30.076105 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 25145
I20260620 18:45:30.076313 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:45:30.076937 15771 external_mini_cluster.cc:1428] Setting key 9412e66c45227ee1ac3e40fcb38acc93
I20260620 18:45:30.079829 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35123
--builtin_ntp_servers=127.15.102.212:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:45:30.080873 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:30.070174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:40345 (local address 127.15.102.254:35123)
0620 18:45:30.070627 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:30.070632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:30.071775 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 18:45:30.073641 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:30.073647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:30.073773 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:30.073961 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:30.075010 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:30.075975 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:30.077812 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:30.078289 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:30.078384 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:30.079927 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:30.079935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:30.079941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:30.079961 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:30.080010 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:30.080030 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:30.080146 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:30.080428 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 18:45:30.080536 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":237,"threads_started":1}
I20260620 18:45:30.081928 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:30.070647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35123 (local address 127.15.102.193:40345)
0620 18:45:30.071622 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:30.071664 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:30.073344 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:30.074196 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:30.074204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:30.074294 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:30.074849 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:30.074865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:30.076135 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:30.076141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:30.077665 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:30.077683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:30.078527 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:30.078535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:30.078865 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:30.079602 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:45:30.079605 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:30.079653 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:45:30.079733 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:30.080196 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:30.080205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:30.080215 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:45:30.080413 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 18:45:30.080804 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":286,"threads_started":1}
I20260620 18:45:30.083560 25272 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35123
I20260620 18:45:30.083869 25272 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:30.084836 25272 heartbeater.cc:511] Master 127.15.102.254:35123 requested a full tablet report, sending...
I20260620 18:45:30.087109 25086 ts_manager.cc:194] Registered new tserver with Master: e1d3b9f3fd3b426ebf1712c6e61b2688 (127.15.102.193:33115)
I20260620 18:45:30.088582 25086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.193:40345
W20260620 18:45:30.404093 25276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:30.404376 25276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:30.404513 25276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:30.411887 25276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:30.411989 25276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:30.412150 25276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:45:30.421386 25276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:30.422717 25276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:30.424500 25276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:30.437377 25281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:30.437345 25284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:30.437304 25282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:30.438071 25276 server_base.cc:1034] running on GCE node
I20260620 18:45:30.438663 25276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:30.439692 25276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:30.440919 25276 hybrid_clock.cc:648] HybridClock initialized: now 1781981130440882 us; error 54 us; skew 500 ppm
I20260620 18:45:30.441241 25276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:45:30.443574 25276 webserver.cc:460] Webserver started at http://127.15.102.194:45653/ using document root <none> and password file <none>
I20260620 18:45:30.444283 25276 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:30.444413 25276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:30.444749 25276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:30.447642 25276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "9c85c771b0044eb9af010ed3f25178fc"
format_stamp: "Formatted at 2026-06-20 18:45:30 on dist-test-slave-zfp8"
server_key: "a0b9be73289ea5e6a1494bdcf4c0f756"
server_key_iv: "ffa00d9fd465483884b9147352660319"
server_key_version: "encryptionkey@0"
I20260620 18:45:30.448426 25276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "9c85c771b0044eb9af010ed3f25178fc"
format_stamp: "Formatted at 2026-06-20 18:45:30 on dist-test-slave-zfp8"
server_key: "a0b9be73289ea5e6a1494bdcf4c0f756"
server_key_iv: "ffa00d9fd465483884b9147352660319"
server_key_version: "encryptionkey@0"
I20260620 18:45:30.454947 25276 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20260620 18:45:30.459373 25290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.460687 25276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:45:30.460813 25276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "9c85c771b0044eb9af010ed3f25178fc"
format_stamp: "Formatted at 2026-06-20 18:45:30 on dist-test-slave-zfp8"
server_key: "a0b9be73289ea5e6a1494bdcf4c0f756"
server_key_iv: "ffa00d9fd465483884b9147352660319"
server_key_version: "encryptionkey@0"
I20260620 18:45:30.460983 25276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:30.487015 25276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:30.487913 25276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:30.488176 25276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:30.489043 25276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:30.490413 25276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:30.490501 25276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.490573 25276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:30.490617 25276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.523547 25276 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:40159
I20260620 18:45:30.523635 25402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:40159 every 8 connection(s)
I20260620 18:45:30.524868 25276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:30.533609 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 25276
I20260620 18:45:30.533773 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:45:30.534314 15771 external_mini_cluster.cc:1428] Setting key 8a93945902b48fcc8b6361f6deeadd7c
I20260620 18:45:30.537269 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35123
--builtin_ntp_servers=127.15.102.212:44767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 18:45:30.541532 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:30.527399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:50091 (local address 127.15.102.254:35123)
0620 18:45:30.527574 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:30.527579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:30.528998 (+ 1419us) server_negotiation.cc:407] Connection header received
0620 18:45:30.530740 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:30.530745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:30.530835 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:30.530959 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:30.532066 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:30.532781 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:30.534995 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:30.535680 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:30.535814 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:30.540672 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:30.540677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:30.540680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:30.540697 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:30.540728 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:30.540739 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:30.540838 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:30.541068 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:45:30.541198 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:45:30.542608 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:30.527950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35123 (local address 127.15.102.194:50091)
0620 18:45:30.528841 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:30.528877 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:30.530458 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:30.531185 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:30.531193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:30.531340 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:30.531903 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:30.531919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:30.532945 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:30.532952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:30.534852 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:30.534867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:30.535928 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:30.535936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:30.536233 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:30.540325 (+ 4092us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:45:30.540327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:30.540380 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:45:30.540464 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:30.540838 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:30.540852 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:30.540862 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:45:30.541043 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 18:45:30.541471 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
I20260620 18:45:30.544081 25403 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35123
I20260620 18:45:30.544404 25403 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:30.545307 25403 heartbeater.cc:511] Master 127.15.102.254:35123 requested a full tablet report, sending...
I20260620 18:45:30.546866 25086 ts_manager.cc:194] Registered new tserver with Master: 9c85c771b0044eb9af010ed3f25178fc (127.15.102.194:40159)
I20260620 18:45:30.548146 25086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.194:50091
W20260620 18:45:30.874361 25406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:30.874629 25406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:30.874709 25406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:30.881800 25406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:30.881898 25406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:30.882001 25406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:45:30.891116 25406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:30.892357 25406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:30.894068 25406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:30.907128 25411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:30.907166 25414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:30.907213 25412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:30.907706 25406 server_base.cc:1034] running on GCE node
I20260620 18:45:30.908602 25406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:30.909469 25406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:30.910688 25406 hybrid_clock.cc:648] HybridClock initialized: now 1781981130910623 us; error 55 us; skew 500 ppm
I20260620 18:45:30.911046 25406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:45:30.913218 25406 webserver.cc:460] Webserver started at http://127.15.102.195:41977/ using document root <none> and password file <none>
I20260620 18:45:30.913980 25406 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:30.914074 25406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:30.914388 25406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:30.917171 25406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "3d23deec8a304e3ab1f0b34839ddbfd4"
format_stamp: "Formatted at 2026-06-20 18:45:30 on dist-test-slave-zfp8"
server_key: "cf5bd9fed040d51033cabc8026e0bbc0"
server_key_iv: "ab127e7b8055ec10856c6d52eb6bbe4e"
server_key_version: "encryptionkey@0"
I20260620 18:45:30.917937 25406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "3d23deec8a304e3ab1f0b34839ddbfd4"
format_stamp: "Formatted at 2026-06-20 18:45:30 on dist-test-slave-zfp8"
server_key: "cf5bd9fed040d51033cabc8026e0bbc0"
server_key_iv: "ab127e7b8055ec10856c6d52eb6bbe4e"
server_key_version: "encryptionkey@0"
I20260620 18:45:30.922889 25406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:45:30.926389 25420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.927772 25406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:30.927946 25406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "3d23deec8a304e3ab1f0b34839ddbfd4"
format_stamp: "Formatted at 2026-06-20 18:45:30 on dist-test-slave-zfp8"
server_key: "cf5bd9fed040d51033cabc8026e0bbc0"
server_key_iv: "ab127e7b8055ec10856c6d52eb6bbe4e"
server_key_version: "encryptionkey@0"
I20260620 18:45:30.928164 25406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:30.955085 25406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:30.956004 25406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:30.956324 25406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:30.957309 25406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:30.958773 25406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:30.958885 25406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.959029 25406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:30.959128 25406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:30.992729 25406 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:33869
I20260620 18:45:30.992825 25532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:33869 every 8 connection(s)
I20260620 18:45:30.994026 25406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:31.002542 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 25406
I20260620 18:45:31.002719 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:45:31.003182 15771 external_mini_cluster.cc:1428] Setting key e571f3d4fa6aff3a19e096aa0cca91ea
I20260620 18:45:31.006779 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:30.996571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:44037 (local address 127.15.102.254:35123)
0620 18:45:30.996761 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:30.996765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:30.998153 (+ 1388us) server_negotiation.cc:407] Connection header received
0620 18:45:30.999899 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:30.999903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:30.999970 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:31.000059 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:31.001052 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.001755 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.003440 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.004102 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.004259 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:31.005942 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:31.005946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:31.005950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:31.005964 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:31.005994 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:31.006005 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:31.006074 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:31.006365 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 18:45:31.006464 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:45:31.007958 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:30.997124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35123 (local address 127.15.102.195:44037)
0620 18:45:30.997972 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:30.998015 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:30.999652 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:31.000271 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:31.000280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:31.000371 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:31.000913 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:31.000928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.001886 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.001890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:31.003271 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:31.003301 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.004372 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.004378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:31.004701 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:31.005610 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:45:31.005614 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:31.005671 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:45:31.005780 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:31.006103 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:31.006112 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:31.006119 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:45:31.006334 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 18:45:31.006735 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":245,"threads_started":1}
I20260620 18:45:31.009305 25533 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35123
I20260620 18:45:31.009611 25533 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:31.010316 25533 heartbeater.cc:511] Master 127.15.102.254:35123 requested a full tablet report, sending...
I20260620 18:45:31.011673 25086 ts_manager.cc:194] Registered new tserver with Master: 3d23deec8a304e3ab1f0b34839ddbfd4 (127.15.102.195:33869)
I20260620 18:45:31.012432 25086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.195:44037
I20260620 18:45:31.018731 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:45:31.024745 25274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:31.023203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.15.102.254:35123)
0620 18:45:31.023429 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:31.023434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:31.023843 (+ 409us) server_negotiation.cc:407] Connection header received
0620 18:45:31.023973 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:31.023977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:31.024045 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:31.024128 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:31.024473 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34904: BlockingRecv error: recv got EOF from 127.0.0.1:34904 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:45:31.025677 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 25145
I20260620 18:45:31.033054 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:31.026501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.15.102.193:33115)
0620 18:45:31.026963 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:31.026971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:31.026997 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:45:31.027058 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:31.027063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:31.027343 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:31.027472 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:31.027785 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.028719 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.029879 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.030407 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.030499 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:31.031469 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:31.031478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:31.031491 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:31.032148 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:31.032193 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:31.032217 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:31.032366 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:31.032515 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 18:45:31.032638 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260620 18:45:31.034219 25267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:31.113926 25272 heartbeater.cc:503] Master 127.15.102.254:35123 was elected leader, sending a full tablet report...
I20260620 18:45:31.115180 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 25145
W20260620 18:45:31.131299 25073 connection.cc:535] server connection from 127.15.102.193:40345 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:45:31.131621 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 25276
I20260620 18:45:31.139165 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:31.132583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.15.102.194:40159)
0620 18:45:31.133159 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:31.133167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:31.133193 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:45:31.133251 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:31.133258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:31.133481 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:31.133607 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:31.133924 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.134896 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.136185 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.136738 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.136849 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:31.137762 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:31.137770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:31.137783 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:31.138378 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:31.138424 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:31.138443 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:31.138545 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:31.138626 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 18:45:31.138728 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":289,"threads_started":1}
I20260620 18:45:31.140482 25398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:31.220896 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 25276
I20260620 18:45:31.238020 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 25406
I20260620 18:45:31.246198 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:31.239054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.15.102.195:33869)
0620 18:45:31.239722 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:31.239729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:31.239758 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:45:31.239813 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:31.239820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:31.240036 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:31.240171 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:31.240585 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.241607 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.242808 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:31.243482 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:31.243609 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:31.244740 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:31.244748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:31.244761 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:31.245339 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:31.245383 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:31.245401 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:31.245588 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:31.245691 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:45:31.245793 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":361,"threads_started":1}
I20260620 18:45:31.247583 25528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:31.329579 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 25406
I20260620 18:45:31.345549 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 25055
I20260620 18:45:31.346815 25116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:31.448124 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 25055
2026-06-20T18:45:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2396 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T18:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:31Z Disabled control of system clock
I20260620 18:45:31.519368 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46407
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:46407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:45:31.845693 25552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:31.845996 25552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:31.846079 25552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:31.853165 25552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:31.853281 25552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:31.853319 25552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:31.853379 25552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:31.853422 25552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:31.862542 25552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:46407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:31.863832 25552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:31.865607 25552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:31.877113 25560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:31.877136 25558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:31.877375 25557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:31.878041 25552 server_base.cc:1034] running on GCE node
I20260620 18:45:31.878794 25552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:31.880240 25552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:31.881433 25552 hybrid_clock.cc:648] HybridClock initialized: now 1781981131881379 us; error 54 us; skew 500 ppm
I20260620 18:45:31.881742 25552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:45:31.883978 25552 webserver.cc:460] Webserver started at http://127.15.102.254:37389/ using document root <none> and password file <none>
I20260620 18:45:31.884703 25552 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:31.884790 25552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:31.885056 25552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:31.887938 25552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "b57f69fa5b224753abed828d619dd809"
format_stamp: "Formatted at 2026-06-20 18:45:31 on dist-test-slave-zfp8"
server_key: "b8dd72a1783e149f008194018b489ea8"
server_key_iv: "06efbb9bdc1e5b38df99740b5701099e"
server_key_version: "encryptionkey@0"
I20260620 18:45:31.888693 25552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "b57f69fa5b224753abed828d619dd809"
format_stamp: "Formatted at 2026-06-20 18:45:31 on dist-test-slave-zfp8"
server_key: "b8dd72a1783e149f008194018b489ea8"
server_key_iv: "06efbb9bdc1e5b38df99740b5701099e"
server_key_version: "encryptionkey@0"
I20260620 18:45:31.893770 25552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:45:31.897146 25566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:31.898481 25552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:45:31.898674 25552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "b57f69fa5b224753abed828d619dd809"
format_stamp: "Formatted at 2026-06-20 18:45:31 on dist-test-slave-zfp8"
server_key: "b8dd72a1783e149f008194018b489ea8"
server_key_iv: "06efbb9bdc1e5b38df99740b5701099e"
server_key_version: "encryptionkey@0"
I20260620 18:45:31.898907 25552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:31.904505 25552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:31.905316 25552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:31.905617 25552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:31.923985 25552 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:46407
I20260620 18:45:31.924008 25617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:46407 every 8 connection(s)
I20260620 18:45:31.925374 25552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:31.929085 25618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:31.930094 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 25552
I20260620 18:45:31.930316 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:45:31.931111 15771 external_mini_cluster.cc:1428] Setting key 92f7588b52143eb52aabbe2ba162b482
I20260620 18:45:31.939687 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:31.933335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.15.102.254:46407)
0620 18:45:31.934419 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:31.934431 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:31.934474 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:45:31.936053 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:31.936076 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:31.936528 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:31.936866 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:31.937942 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:31.937948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:31.937960 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:31.937975 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:31.938035 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:31.938052 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:31.938240 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:31.938362 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:45:31.938671 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":412,"threads_started":1}
I20260620 18:45:31.942747 25618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809: Bootstrap starting.
I20260620 18:45:31.945685 25618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:31.946657 25618 log.cc:824] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:31.949424 25618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809: No bootstrap required, opened a new log
I20260620 18:45:31.956645 25618 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f69fa5b224753abed828d619dd809" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46407 } }
I20260620 18:45:31.956995 25618 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:31.957119 25618 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57f69fa5b224753abed828d619dd809, State: Initialized, Role: FOLLOWER
I20260620 18:45:31.957755 25618 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f69fa5b224753abed828d619dd809" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46407 } }
I20260620 18:45:31.957983 25618 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:31.958118 25618 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:31.958280 25618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:31.960525 25618 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f69fa5b224753abed828d619dd809" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46407 } }
I20260620 18:45:31.961066 25618 leader_election.cc:302] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57f69fa5b224753abed828d619dd809; no voters:
I20260620 18:45:31.961670 25618 leader_election.cc:288] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:45:31.961817 25623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:31.962531 25623 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [term 1 LEADER]: Becoming Leader. State: Replica: b57f69fa5b224753abed828d619dd809, State: Running, Role: LEADER
I20260620 18:45:31.963156 25623 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f69fa5b224753abed828d619dd809" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46407 } }
I20260620 18:45:31.963861 25618 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:31.965783 25624 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57f69fa5b224753abed828d619dd809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f69fa5b224753abed828d619dd809" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46407 } } }
I20260620 18:45:31.966097 25624 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:31.968252 25625 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57f69fa5b224753abed828d619dd809. Latest consensus state: current_term: 1 leader_uuid: "b57f69fa5b224753abed828d619dd809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57f69fa5b224753abed828d619dd809" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46407 } } }
I20260620 18:45:31.968526 25625 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809 [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:31.968897 25629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:31.973666 25629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:31.982261 25629 catalog_manager.cc:1347] Generated new cluster ID: 2bb9cb10037e4026b233802d70e1e661
I20260620 18:45:31.982386 25629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:31.996346 25629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:45:31.996843 25629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:32.004648 25629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57f69fa5b224753abed828d619dd809: Generated new TSK 0
I20260620 18:45:32.005534 25629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:45:32.012789 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46407
--builtin_ntp_servers=127.15.102.212:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 18:45:32.347550 25642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:32.347900 25642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:32.348073 25642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:32.355419 25642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:32.355568 25642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:32.355770 25642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:45:32.365236 25642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:32.366693 25642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:32.368531 25642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:32.382067 25650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:32.382144 25647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:32.382136 25648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:32.382889 25642 server_base.cc:1034] running on GCE node
I20260620 18:45:32.383539 25642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:32.384485 25642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:32.385690 25642 hybrid_clock.cc:648] HybridClock initialized: now 1781981132385637 us; error 44 us; skew 500 ppm
I20260620 18:45:32.386019 25642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:45:32.388458 25642 webserver.cc:460] Webserver started at http://127.15.102.193:42635/ using document root <none> and password file <none>
I20260620 18:45:32.389175 25642 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:32.389287 25642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:32.389611 25642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:32.392535 25642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "0af58777d6224fc8936e62bb14b0ac37"
format_stamp: "Formatted at 2026-06-20 18:45:32 on dist-test-slave-zfp8"
server_key: "c89eebedb226218bc90bfd427d77fb28"
server_key_iv: "7b0ed401efeaff68618904029107ac0b"
server_key_version: "encryptionkey@0"
I20260620 18:45:32.393347 25642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "0af58777d6224fc8936e62bb14b0ac37"
format_stamp: "Formatted at 2026-06-20 18:45:32 on dist-test-slave-zfp8"
server_key: "c89eebedb226218bc90bfd427d77fb28"
server_key_iv: "7b0ed401efeaff68618904029107ac0b"
server_key_version: "encryptionkey@0"
I20260620 18:45:32.398630 25642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 18:45:32.402263 25656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:32.403750 25642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:32.403919 25642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "0af58777d6224fc8936e62bb14b0ac37"
format_stamp: "Formatted at 2026-06-20 18:45:32 on dist-test-slave-zfp8"
server_key: "c89eebedb226218bc90bfd427d77fb28"
server_key_iv: "7b0ed401efeaff68618904029107ac0b"
server_key_version: "encryptionkey@0"
I20260620 18:45:32.404162 25642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:32.409859 25642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:32.410662 25642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:32.410938 25642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:32.411944 25642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:32.413311 25642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:32.413420 25642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:32.413549 25642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:32.413651 25642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:32.445739 25642 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:33731
I20260620 18:45:32.445786 25768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:33731 every 8 connection(s)
I20260620 18:45:32.447054 25642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:32.448096 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 25642
I20260620 18:45:32.448304 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:45:32.449021 15771 external_mini_cluster.cc:1428] Setting key e2b4c1c7980c0ba1e321d768575dd102
I20260620 18:45:32.452275 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46407
--builtin_ntp_servers=127.15.102.212:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:45:32.462569 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:32.449857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:59889 (local address 127.15.102.254:46407)
0620 18:45:32.450293 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:32.450298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:32.459304 (+ 9006us) server_negotiation.cc:407] Connection header received
0620 18:45:32.459629 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:32.459638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:32.459715 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:32.459821 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:32.461595 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:32.461630 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:32.461635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:32.461661 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:32.461747 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:32.461783 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:32.461921 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:32.462124 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 18:45:32.462194 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":230,"threads_started":1}
I20260620 18:45:32.463480 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:32.450511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46407 (local address 127.15.102.193:59889)
0620 18:45:32.456499 (+ 5988us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:32.456561 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:32.459138 (+ 2577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:32.460135 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:32.460150 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:32.460295 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:32.461187 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:45:32.461191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:32.461249 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:45:32.461372 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:32.461909 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:32.461918 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:32.461924 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:45:32.462164 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 18:45:32.462387 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5085,"thread_start_us":4816,"threads_started":1}
I20260620 18:45:32.465201 25769 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46407
I20260620 18:45:32.465548 25769 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:32.466132 25769 heartbeater.cc:511] Master 127.15.102.254:46407 requested a full tablet report, sending...
I20260620 18:45:32.468135 25583 ts_manager.cc:194] Registered new tserver with Master: 0af58777d6224fc8936e62bb14b0ac37 (127.15.102.193:33731)
W20260620 18:45:32.781883 25773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:32.782187 25773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:32.782279 25773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:32.789587 25773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:32.789693 25773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:32.789801 25773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:45:32.798864 25773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:32.800223 25773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:32.801908 25773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:32.815485 25779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:32.815549 25778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:32.815553 25781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:32.816571 25773 server_base.cc:1034] running on GCE node
I20260620 18:45:32.817106 25773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:32.818096 25773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:32.819329 25773 hybrid_clock.cc:648] HybridClock initialized: now 1781981132819278 us; error 33 us; skew 500 ppm
I20260620 18:45:32.819618 25773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:45:32.821919 25773 webserver.cc:460] Webserver started at http://127.15.102.194:34569/ using document root <none> and password file <none>
I20260620 18:45:32.822564 25773 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:32.822656 25773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:32.822909 25773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:32.827613 25773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "242f268ceb30479ba10e9a6fa393b17e"
format_stamp: "Formatted at 2026-06-20 18:45:32 on dist-test-slave-zfp8"
server_key: "71e498b1afa87d759fa836a32b53778e"
server_key_iv: "10b0ad332608039bbdbda001b510ff4c"
server_key_version: "encryptionkey@0"
I20260620 18:45:32.828361 25773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "242f268ceb30479ba10e9a6fa393b17e"
format_stamp: "Formatted at 2026-06-20 18:45:32 on dist-test-slave-zfp8"
server_key: "71e498b1afa87d759fa836a32b53778e"
server_key_iv: "10b0ad332608039bbdbda001b510ff4c"
server_key_version: "encryptionkey@0"
I20260620 18:45:32.833343 25773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:45:32.836617 25787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:32.838300 25773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:45:32.838441 25773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "242f268ceb30479ba10e9a6fa393b17e"
format_stamp: "Formatted at 2026-06-20 18:45:32 on dist-test-slave-zfp8"
server_key: "71e498b1afa87d759fa836a32b53778e"
server_key_iv: "10b0ad332608039bbdbda001b510ff4c"
server_key_version: "encryptionkey@0"
I20260620 18:45:32.838644 25773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:32.844275 25773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:32.845036 25773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:32.845254 25773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:32.846109 25773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:32.847527 25773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:32.847620 25773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:32.847702 25773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:32.847775 25773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:32.880980 25773 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:45263
I20260620 18:45:32.881072 25899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:45263 every 8 connection(s)
I20260620 18:45:32.882300 25773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:32.889014 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 25773
I20260620 18:45:32.889217 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:45:32.889757 15771 external_mini_cluster.cc:1428] Setting key 5bceb29b8582575fb5821c8901795da4
I20260620 18:45:32.891455 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:32.885112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:60389 (local address 127.15.102.254:46407)
0620 18:45:32.885296 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:32.885300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:32.886744 (+ 1444us) server_negotiation.cc:407] Connection header received
0620 18:45:32.888538 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:32.888542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:32.888609 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:32.888719 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:32.890626 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:32.890630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:32.890633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:32.890659 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:32.890689 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:32.890700 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:32.890832 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:32.891040 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:45:32.891106 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:45:32.892755 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46407
--builtin_ntp_servers=127.15.102.212:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 18:45:32.892784 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:32.885637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46407 (local address 127.15.102.194:60389)
0620 18:45:32.886565 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:32.886605 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:32.888207 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:32.889125 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:32.889139 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:32.889278 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:32.890073 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:45:32.890077 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:32.890129 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:45:32.890245 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:32.890808 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:32.890820 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:32.890828 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:45:32.891125 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 18:45:32.891438 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":287,"threads_started":1}
I20260620 18:45:32.894170 25900 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46407
I20260620 18:45:32.894522 25900 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:32.895190 25900 heartbeater.cc:511] Master 127.15.102.254:46407 requested a full tablet report, sending...
I20260620 18:45:32.896759 25583 ts_manager.cc:194] Registered new tserver with Master: 242f268ceb30479ba10e9a6fa393b17e (127.15.102.194:45263)
W20260620 18:45:33.231051 25903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:33.231354 25903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:33.231509 25903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:33.238715 25903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:33.238814 25903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:33.239015 25903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:45:33.248198 25903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:33.249514 25903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:33.251307 25903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:33.264209 25908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:33.264285 25911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:33.264523 25909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:33.265570 25903 server_base.cc:1034] running on GCE node
I20260620 18:45:33.266188 25903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:33.267171 25903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:33.268455 25903 hybrid_clock.cc:648] HybridClock initialized: now 1781981133268397 us; error 45 us; skew 500 ppm
I20260620 18:45:33.268795 25903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:45:33.271083 25903 webserver.cc:460] Webserver started at http://127.15.102.195:46035/ using document root <none> and password file <none>
I20260620 18:45:33.271865 25903 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:33.272001 25903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:33.272324 25903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:33.275122 25903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "2bd95384f88b4d0e88cd5606e4a42d54"
format_stamp: "Formatted at 2026-06-20 18:45:33 on dist-test-slave-zfp8"
server_key: "564897123af9ebe28e85fadb41573804"
server_key_iv: "d7787232ffb6f5a0cdc9089967d9511e"
server_key_version: "encryptionkey@0"
I20260620 18:45:33.275969 25903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "2bd95384f88b4d0e88cd5606e4a42d54"
format_stamp: "Formatted at 2026-06-20 18:45:33 on dist-test-slave-zfp8"
server_key: "564897123af9ebe28e85fadb41573804"
server_key_iv: "d7787232ffb6f5a0cdc9089967d9511e"
server_key_version: "encryptionkey@0"
I20260620 18:45:33.281054 25903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:45:33.284549 25917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:33.285925 25903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:33.286223 25903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "2bd95384f88b4d0e88cd5606e4a42d54"
format_stamp: "Formatted at 2026-06-20 18:45:33 on dist-test-slave-zfp8"
server_key: "564897123af9ebe28e85fadb41573804"
server_key_iv: "d7787232ffb6f5a0cdc9089967d9511e"
server_key_version: "encryptionkey@0"
I20260620 18:45:33.286473 25903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:33.292303 25903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:33.293098 25903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:33.293373 25903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:33.294255 25903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:33.295691 25903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:33.295814 25903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:33.295946 25903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:33.296034 25903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:33.328658 25903 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:39175
I20260620 18:45:33.328712 26029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:39175 every 8 connection(s)
I20260620 18:45:33.330029 25903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:33.336557 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 25903
I20260620 18:45:33.336740 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:45:33.337181 15771 external_mini_cluster.cc:1428] Setting key 7c62bd3810d3c1c8a4afd0f16b7d122e
I20260620 18:45:33.338613 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:33.332613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:34599 (local address 127.15.102.254:46407)
0620 18:45:33.332789 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:33.332793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:33.334337 (+ 1544us) server_negotiation.cc:407] Connection header received
0620 18:45:33.336112 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:33.336118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:33.336188 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:33.336305 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:33.337689 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:33.337695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:33.337699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:33.337718 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:33.337758 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:33.337770 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:33.337905 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:33.338279 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 18:45:33.338343 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:45:33.339620 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:33.333214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46407 (local address 127.15.102.195:34599)
0620 18:45:33.334160 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:33.334203 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:33.335838 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:33.336527 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:33.336537 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:33.336655 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:33.337397 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:45:33.337400 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:33.337436 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:45:33.337521 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:33.338082 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:33.338090 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:33.338097 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:45:33.338361 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 18:45:33.338616 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":292,"threads_started":1}
I20260620 18:45:33.340917 26030 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46407
I20260620 18:45:33.341181 26030 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:33.341703 26030 heartbeater.cc:511] Master 127.15.102.254:46407 requested a full tablet report, sending...
I20260620 18:45:33.343034 25583 ts_manager.cc:194] Registered new tserver with Master: 2bd95384f88b4d0e88cd5606e4a42d54 (127.15.102.195:39175)
I20260620 18:45:33.352766 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 18:45:33.359072 25771 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:45:33.357440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.15.102.254:46407)
0620 18:45:33.357603 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:33.357607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:33.358075 (+ 468us) server_negotiation.cc:407] Connection header received
0620 18:45:33.358268 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:33.358274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:33.358333 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:33.358437 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:33.358781 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50598: BlockingRecv error: recv got EOF from 127.0.0.1:50598 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:45:33.359732 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 25642
I20260620 18:45:33.363406 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:33.360700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.15.102.193:33731)
0620 18:45:33.361234 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:33.361240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:33.361265 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:45:33.361319 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:33.361324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:33.361535 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:33.361696 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:33.362667 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:33.362673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:33.362684 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:33.362700 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:33.362738 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:33.362751 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:33.362892 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:33.362995 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 18:45:33.363094 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
I20260620 18:45:33.364679 25764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:33.445286 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 25642
I20260620 18:45:33.461473 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 25773
I20260620 18:45:33.465200 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:33.462503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.15.102.194:45263)
0620 18:45:33.462983 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:33.462990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:33.463017 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:45:33.463079 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:33.463086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:33.463386 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:33.463516 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:33.464480 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:33.464487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:33.464499 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:33.464511 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:33.464566 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:33.464592 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:33.464698 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:33.464870 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 18:45:33.464976 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":228,"threads_started":1}
I20260620 18:45:33.466434 25895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:33.548492 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 25773
I20260620 18:45:33.566561 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 25903
I20260620 18:45:33.570456 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:33.567577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.15.102.195:39175)
0620 18:45:33.568245 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:33.568252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:33.568282 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:45:33.568341 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:33.568349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:33.568569 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:33.568697 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:33.569734 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:33.569740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:33.569752 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:45:33.569767 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:33.569806 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:45:33.569819 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:33.569947 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:33.570009 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 18:45:33.570127 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":342,"threads_started":1}
I20260620 18:45:33.571815 26025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:33.653086 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 25903
I20260620 18:45:33.669271 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 25552
I20260620 18:45:33.670485 25613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:33.770026 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 25552
2026-06-20T18:45:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2322 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:45:33 dist-test-slave-zfp8 krb5kdc[26048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:45:33 dist-test-slave-zfp8 krb5kdc[26048](info): set up 2 sockets
Jun 20 18:45:33 dist-test-slave-zfp8 krb5kdc[26048](info): commencing operation
krb5kdc: starting...
W20260620 18:45:34.896291 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:45:35 dist-test-slave-zfp8 krb5kdc[26048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:45:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:45:35.120532 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35015
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:35015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:35.442728 26061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:35.443050 26061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:35.443217 26061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:35.450826 26061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:45:35.450948 26061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:35.451028 26061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:35.451134 26061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:45:35.451218 26061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:45:35.461476 26061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:35015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:35.463119 26061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:35.464536 26061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:35.479703 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:35.479732 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:35.479758 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:35.480528 26061 server_base.cc:1034] running on GCE node
I20260620 18:45:35.481390 26061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:35.482988 26061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:35.484221 26061 hybrid_clock.cc:648] HybridClock initialized: now 1781981135484166 us; error 56 us; skew 500 ppm
I20260620 18:45:35.484589 26061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:35 dist-test-slave-zfp8 krb5kdc[26048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981135, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:35.487823 26061 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:45:35.489454 26061 webserver.cc:460] Webserver started at http://127.15.102.254:41811/ using document root <none> and password file <none>
I20260620 18:45:35.490175 26061 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:35.490325 26061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:35.490639 26061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:35.494899 26061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "d447525447a74471a10bdceec4b1eb9b"
format_stamp: "Formatted at 2026-06-20 18:45:35 on dist-test-slave-zfp8"
server_key: "1d0f2b7a2b9e91aef2e577e166be1c76"
server_key_iv: "a71fd6acfea4b40c1e0390d36ad6b9bb"
server_key_version: "encryptionkey@0"
I20260620 18:45:35.495749 26061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "d447525447a74471a10bdceec4b1eb9b"
format_stamp: "Formatted at 2026-06-20 18:45:35 on dist-test-slave-zfp8"
server_key: "1d0f2b7a2b9e91aef2e577e166be1c76"
server_key_iv: "a71fd6acfea4b40c1e0390d36ad6b9bb"
server_key_version: "encryptionkey@0"
I20260620 18:45:35.501088 26061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:45:35.504551 26076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:35.505923 26061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:35.506131 26061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "d447525447a74471a10bdceec4b1eb9b"
format_stamp: "Formatted at 2026-06-20 18:45:35 on dist-test-slave-zfp8"
server_key: "1d0f2b7a2b9e91aef2e577e166be1c76"
server_key_iv: "a71fd6acfea4b40c1e0390d36ad6b9bb"
server_key_version: "encryptionkey@0"
I20260620 18:45:35.506331 26061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:35.575383 26061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:35.579527 26061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:35.579823 26061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:35.598160 26061 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:35015
I20260620 18:45:35.598192 26127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:35015 every 8 connection(s)
I20260620 18:45:35.599601 26061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:45:35.603363 26128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:35.604216 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 26061
I20260620 18:45:35.604470 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:45:35.605232 15771 external_mini_cluster.cc:1428] Setting key 3725015001b4bb84d8cf5dcb4c94365c
I20260620 18:45:35.617583 26128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b: Bootstrap starting.
Jun 20 18:45:35 dist-test-slave-zfp8 krb5kdc[26048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:45:35.621320 26128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:35.622399 26128 log.cc:824] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:35.625779 26128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b: No bootstrap required, opened a new log
I20260620 18:45:35.629556 26131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:35.607418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.15.102.254:35015)
0620 18:45:35.608220 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:35.608234 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:35.608278 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:45:35.609937 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:35.609998 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:35.610499 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:35.610868 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:35.612619 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:35.613921 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:35.615726 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:35.616366 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:35.616497 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:35.621075 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:35.621082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:35.621097 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:35.621135 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:35.624855 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:35.625558 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:35.625566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:35.625574 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:35.625785 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:35.626246 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:35.626252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:35.626257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:35.626952 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:35.627272 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:35.627679 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 18:45:35.628141 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":377,"threads_started":1}
I20260620 18:45:35.635013 26128 raft_consensus.cc:348] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d447525447a74471a10bdceec4b1eb9b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35015 } }
I20260620 18:45:35.635406 26128 raft_consensus.cc:374] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:35.635483 26128 raft_consensus.cc:729] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d447525447a74471a10bdceec4b1eb9b, State: Initialized, Role: FOLLOWER
I20260620 18:45:35.636016 26128 consensus_queue.cc:260] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d447525447a74471a10bdceec4b1eb9b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35015 } }
I20260620 18:45:35.636206 26128 raft_consensus.cc:388] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:45:35.636296 26128 raft_consensus.cc:482] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:45:35.636418 26128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:35.638734 26128 raft_consensus.cc:504] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d447525447a74471a10bdceec4b1eb9b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35015 } }
I20260620 18:45:35.639199 26128 leader_election.cc:302] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d447525447a74471a10bdceec4b1eb9b; no voters:
I20260620 18:45:35.639755 26128 leader_election.cc:288] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:45:35.639884 26133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:35.640642 26133 raft_consensus.cc:686] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [term 1 LEADER]: Becoming Leader. State: Replica: d447525447a74471a10bdceec4b1eb9b, State: Running, Role: LEADER
I20260620 18:45:35.641145 26133 consensus_queue.cc:237] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d447525447a74471a10bdceec4b1eb9b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35015 } }
I20260620 18:45:35.641878 26128 sys_catalog.cc:564] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:45:35.644039 26135 sys_catalog.cc:455] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d447525447a74471a10bdceec4b1eb9b. Latest consensus state: current_term: 1 leader_uuid: "d447525447a74471a10bdceec4b1eb9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d447525447a74471a10bdceec4b1eb9b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35015 } } }
I20260620 18:45:35.644469 26135 sys_catalog.cc:458] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:35.644351 26134 sys_catalog.cc:455] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d447525447a74471a10bdceec4b1eb9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d447525447a74471a10bdceec4b1eb9b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35015 } } }
I20260620 18:45:35.644687 26134 sys_catalog.cc:458] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b [sys.catalog]: This master's current role is: LEADER
I20260620 18:45:35.646791 26142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:45:35.651783 26142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:45:35.660023 26142 catalog_manager.cc:1347] Generated new cluster ID: f6b00e3f2ff440ab82832263258688a5
I20260620 18:45:35.660142 26142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:45:35.678751 26142 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:45:35.680197 26142 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:45:35.694037 26142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b: Generated new TSK 0
I20260620 18:45:35.694921 26142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:45:35.750977 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35015
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:36.079758 26154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:36.080057 26154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:36.080228 26154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:36.087500 26154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:36.087636 26154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:36.087818 26154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:45:36.097611 26154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:36.099036 26154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:36.100437 26154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:36.113595 26160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:36.113559 26162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:36.113638 26159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:36.116727 26154 server_base.cc:1034] running on GCE node
I20260620 18:45:36.117375 26154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:36.118331 26154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:36.119535 26154 hybrid_clock.cc:648] HybridClock initialized: now 1781981136119489 us; error 37 us; skew 500 ppm
I20260620 18:45:36.119854 26154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:36 dist-test-slave-zfp8 krb5kdc[26048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981136, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:36.122891 26154 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:45:36.124454 26154 webserver.cc:460] Webserver started at http://127.15.102.193:35489/ using document root <none> and password file <none>
I20260620 18:45:36.125170 26154 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:36.125313 26154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:36.125734 26154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:36.128666 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "c30a795904f943f4a1317dad319ff07a"
format_stamp: "Formatted at 2026-06-20 18:45:36 on dist-test-slave-zfp8"
server_key: "36b2f26a64d12af6acf07950ad9a1e8f"
server_key_iv: "beb79ef9288687d075829690c633ffd0"
server_key_version: "encryptionkey@0"
I20260620 18:45:36.129436 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "c30a795904f943f4a1317dad319ff07a"
format_stamp: "Formatted at 2026-06-20 18:45:36 on dist-test-slave-zfp8"
server_key: "36b2f26a64d12af6acf07950ad9a1e8f"
server_key_iv: "beb79ef9288687d075829690c633ffd0"
server_key_version: "encryptionkey@0"
I20260620 18:45:36.134651 26154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:45:36.138062 26169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:36.139484 26154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:36.139658 26154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "c30a795904f943f4a1317dad319ff07a"
format_stamp: "Formatted at 2026-06-20 18:45:36 on dist-test-slave-zfp8"
server_key: "36b2f26a64d12af6acf07950ad9a1e8f"
server_key_iv: "beb79ef9288687d075829690c633ffd0"
server_key_version: "encryptionkey@0"
I20260620 18:45:36.139854 26154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:36.182643 26154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:36.186586 26154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:36.186913 26154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:36.187886 26154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:36.189253 26154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:36.189365 26154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:36.189513 26154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:36.189586 26154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:36.222388 26154 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:44597
I20260620 18:45:36.222435 26281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:44597 every 8 connection(s)
I20260620 18:45:36.223704 26154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:45:36.233774 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 26154
I20260620 18:45:36.233997 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:45:36.234704 15771 external_mini_cluster.cc:1428] Setting key 1c98d8404efb00dc86da537a87b034a5
Jun 20 18:45:36 dist-test-slave-zfp8 krb5kdc[26048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981136, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:45:36.243584 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:36.226320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:43075 (local address 127.15.102.254:35015)
0620 18:45:36.226709 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:36.226713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:36.227782 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 18:45:36.229484 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:36.229488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:36.229559 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:36.229658 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:36.231072 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:36.231835 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:36.233394 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:36.234057 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:36.234196 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:36.237868 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:36.237873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:36.237877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:36.237907 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:36.240480 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:36.241216 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:36.241223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:36.241227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:36.241344 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:36.241805 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:36.241810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:36.241813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:36.242129 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:36.242283 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:36.242979 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 18:45:36.243191 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":193,"threads_started":1}
I20260620 18:45:36.244777 26285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:36.226912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35015 (local address 127.15.102.193:43075)
0620 18:45:36.227630 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:36.227667 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:36.229221 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:36.229860 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:36.229868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:36.230379 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:36.230912 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:36.230928 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:36.232005 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:36.232009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:36.233236 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:36.233247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:36.234274 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:36.234279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:36.234485 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:36.235151 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:36.235173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:36.237606 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:36.240637 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:36.240646 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:36.240665 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:36.241094 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:36.241467 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:36.241472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:36.241475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:36.241685 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:36.242267 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:36.242275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:36.242617 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 18:45:36.242902 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 18:45:36.243499 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":300,"threads_started":1}
I20260620 18:45:36.246393 26282 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35015
I20260620 18:45:36.246719 26282 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:36.247620 26282 heartbeater.cc:511] Master 127.15.102.254:35015 requested a full tablet report, sending...
I20260620 18:45:36.249923 26093 ts_manager.cc:194] Registered new tserver with Master: c30a795904f943f4a1317dad319ff07a (127.15.102.193:44597)
I20260620 18:45:36.252049 26093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:43075
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:45:36.286486 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35015
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:36.621825 26288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:36.622109 26288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:36.622202 26288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:36.630038 26288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:36.630146 26288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:36.630259 26288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:45:36.640044 26288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:36.641332 26288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:36.642654 26288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:36.656188 26293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:36.656229 26294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:36.656266 26296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:36.657049 26288 server_base.cc:1034] running on GCE node
I20260620 18:45:36.657600 26288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:36.658573 26288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:36.659801 26288 hybrid_clock.cc:648] HybridClock initialized: now 1781981136659735 us; error 60 us; skew 500 ppm
I20260620 18:45:36.660145 26288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:36 dist-test-slave-zfp8 krb5kdc[26048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981136, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:36.663293 26288 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:45:36.664705 26288 webserver.cc:460] Webserver started at http://127.15.102.194:34627/ using document root <none> and password file <none>
I20260620 18:45:36.665406 26288 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:36.665551 26288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:36.665860 26288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:36.668874 26288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "458d3a53ac0c4e0d9ec10008ce3379b2"
format_stamp: "Formatted at 2026-06-20 18:45:36 on dist-test-slave-zfp8"
server_key: "795510af180f23c03bb616a879148955"
server_key_iv: "dad0e1e02db0e2aab842bde1870b3a55"
server_key_version: "encryptionkey@0"
I20260620 18:45:36.669659 26288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "458d3a53ac0c4e0d9ec10008ce3379b2"
format_stamp: "Formatted at 2026-06-20 18:45:36 on dist-test-slave-zfp8"
server_key: "795510af180f23c03bb616a879148955"
server_key_iv: "dad0e1e02db0e2aab842bde1870b3a55"
server_key_version: "encryptionkey@0"
I20260620 18:45:36.674690 26288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:45:36.678325 26303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:36.679804 26288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:45:36.680019 26288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "458d3a53ac0c4e0d9ec10008ce3379b2"
format_stamp: "Formatted at 2026-06-20 18:45:36 on dist-test-slave-zfp8"
server_key: "795510af180f23c03bb616a879148955"
server_key_iv: "dad0e1e02db0e2aab842bde1870b3a55"
server_key_version: "encryptionkey@0"
I20260620 18:45:36.680250 26288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:36.711511 26288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:36.715801 26288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:36.716120 26288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:36.717111 26288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:36.718523 26288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:36.718639 26288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:36.718783 26288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:36.718863 26288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:36.752040 26288 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:40489
I20260620 18:45:36.752087 26415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:40489 every 8 connection(s)
I20260620 18:45:36.753371 26288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:45:36.758740 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 26288
I20260620 18:45:36.758939 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:45:36.759490 15771 external_mini_cluster.cc:1428] Setting key 537f3a85322509ea119c3c82533ea37f
Jun 20 18:45:36 dist-test-slave-zfp8 krb5kdc[26048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981136, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:45:36.778420 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:36.756023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:34467 (local address 127.15.102.254:35015)
0620 18:45:36.756478 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:36.756483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:36.757452 (+ 969us) server_negotiation.cc:407] Connection header received
0620 18:45:36.759828 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:36.759834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:36.759928 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:36.760041 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:36.762022 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:36.763144 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:36.765410 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:36.766132 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:36.766270 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:36.770683 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:36.770688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:36.770691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:36.770722 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:36.774471 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:36.775577 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:36.775585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:36.775589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:36.775751 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:36.776261 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:36.776268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:36.776272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:36.776673 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:36.776820 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:36.777823 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0620 18:45:36.778034 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":230,"threads_started":1}
I20260620 18:45:36.779440 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:36.756585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35015 (local address 127.15.102.194:34467)
0620 18:45:36.757284 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:36.757322 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:36.759407 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:36.760385 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:36.760394 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:36.761118 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:36.761843 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:36.761862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:36.763337 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:36.763345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:36.765249 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:36.765265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:36.766345 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:36.766353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:36.766650 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:36.767458 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:36.767489 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:36.770431 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:36.774655 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:36.774666 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:36.774683 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:36.775465 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:36.775859 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:36.775865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:36.775871 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:36.776143 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:36.776795 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:36.776803 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:36.777310 (+ 507us) client_negotiation.cc:769] Sending connection context
0620 18:45:36.777618 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 18:45:36.778150 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":287,"threads_started":1}
I20260620 18:45:36.780808 26416 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35015
I20260620 18:45:36.781131 26416 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:36.781867 26416 heartbeater.cc:511] Master 127.15.102.254:35015 requested a full tablet report, sending...
I20260620 18:45:36.783387 26093 ts_manager.cc:194] Registered new tserver with Master: 458d3a53ac0c4e0d9ec10008ce3379b2 (127.15.102.194:40489)
I20260620 18:45:36.784307 26093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:34467
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:45:36.813910 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35015
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:45:37.143729 26422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:45:37.144085 26422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:45:37.144237 26422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:45:37.151829 26422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:45:37.151932 26422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:45:37.152045 26422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:45:37.161784 26422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:45:37.163046 26422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:45:37.164309 26422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:45:37.177312 26428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:37.177390 26427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:45:37.177413 26430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:45:37.178445 26422 server_base.cc:1034] running on GCE node
I20260620 18:45:37.179082 26422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:45:37.180058 26422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:45:37.181284 26422 hybrid_clock.cc:648] HybridClock initialized: now 1781981137181238 us; error 44 us; skew 500 ppm
I20260620 18:45:37.181604 26422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:45:37 dist-test-slave-zfp8 krb5kdc[26048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981137, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:45:37.184933 26422 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:45:37.186424 26422 webserver.cc:460] Webserver started at http://127.15.102.195:43473/ using document root <none> and password file <none>
I20260620 18:45:37.187119 26422 fs_manager.cc:359] Metadata directory not provided
I20260620 18:45:37.187287 26422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:45:37.187618 26422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:45:37.190469 26422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "a81dd2ece1b844eb853d1303aec39757"
format_stamp: "Formatted at 2026-06-20 18:45:37 on dist-test-slave-zfp8"
server_key: "7242b5408d8144b5d0dc967e7efb8ee7"
server_key_iv: "a9e132e8d6046d3ce691587c6decb154"
server_key_version: "encryptionkey@0"
I20260620 18:45:37.191306 26422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "a81dd2ece1b844eb853d1303aec39757"
format_stamp: "Formatted at 2026-06-20 18:45:37 on dist-test-slave-zfp8"
server_key: "7242b5408d8144b5d0dc967e7efb8ee7"
server_key_iv: "a9e132e8d6046d3ce691587c6decb154"
server_key_version: "encryptionkey@0"
I20260620 18:45:37.196446 26422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:45:37.199787 26437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:37.201122 26422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:45:37.201299 26422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "a81dd2ece1b844eb853d1303aec39757"
format_stamp: "Formatted at 2026-06-20 18:45:37 on dist-test-slave-zfp8"
server_key: "7242b5408d8144b5d0dc967e7efb8ee7"
server_key_iv: "a9e132e8d6046d3ce691587c6decb154"
server_key_version: "encryptionkey@0"
I20260620 18:45:37.201493 26422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:45:37.256011 26282 heartbeater.cc:503] Master 127.15.102.254:35015 was elected leader, sending a full tablet report...
I20260620 18:45:37.259812 26422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:45:37.263727 26422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:45:37.263968 26422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:45:37.264833 26422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:45:37.266054 26422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:45:37.266124 26422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:37.266276 26422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:45:37.266324 26422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:45:37.298422 26422 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:38721
I20260620 18:45:37.298465 26549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:38721 every 8 connection(s)
I20260620 18:45:37.299785 26422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:45:37.307113 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 26422
I20260620 18:45:37.307356 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:45:37.307770 15771 external_mini_cluster.cc:1428] Setting key 58689f6aa7ab6e9ffaf6bc5454d1a4cd
Jun 20 18:45:37 dist-test-slave-zfp8 krb5kdc[26048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981137, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:45:37.319814 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.302344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:40981 (local address 127.15.102.254:35015)
0620 18:45:37.302808 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.302815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.303848 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 18:45:37.305602 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.305606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.305678 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.305768 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:37.307298 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.308164 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.310272 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.310799 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.310928 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.314238 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:37.314248 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:37.314251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:37.314280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:37.316615 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.317292 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.317299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.317303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.317424 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.318037 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.318043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.318047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.318318 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:37.318501 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.319170 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.319437 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":213,"threads_started":1}
I20260620 18:45:37.320706 26553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.302988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35015 (local address 127.15.102.195:40981)
0620 18:45:37.303687 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.303732 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:37.305372 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:37.306019 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:37.306028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:37.306575 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:45:37.307117 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.307134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.308329 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.308341 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.310131 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.310143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.311018 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.311026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.311317 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.311987 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:45:37.312008 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:45:37.314039 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:45:37.316763 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:37.316772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:37.316788 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:37.317175 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:37.317700 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:45:37.317707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:45:37.317711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:45:37.317919 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:45:37.318472 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:45:37.318480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:45:37.318818 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 18:45:37.319080 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 18:45:37.319628 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":308,"threads_started":1}
I20260620 18:45:37.322116 26550 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35015
I20260620 18:45:37.322420 26550 heartbeater.cc:464] Registering TS with master...
I20260620 18:45:37.323107 26550 heartbeater.cc:511] Master 127.15.102.254:35015 requested a full tablet report, sending...
I20260620 18:45:37.324529 26092 ts_manager.cc:194] Registered new tserver with Master: a81dd2ece1b844eb853d1303aec39757 (127.15.102.195:38721)
I20260620 18:45:37.325423 26092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:40981
I20260620 18:45:37.335095 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:45:37.351897 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.339747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.15.102.254:35015)
0620 18:45:37.339969 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.339976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.340307 (+ 331us) server_negotiation.cc:407] Connection header received
0620 18:45:37.340500 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.340506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.340568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.340664 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:37.342013 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.342727 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.344067 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.344455 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.344607 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.346921 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:37.346925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:37.346929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:37.346959 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:37.349319 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.349987 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.349994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.349998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.350117 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.350470 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.350477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.350480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.350771 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:37.351036 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.351383 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.351595 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:45:37.356133 26092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:45:37.362888 26092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:45:37.390444 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.375085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:38721 (local address 127.0.0.1:43800)
0620 18:45:37.377329 (+ 2244us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.377385 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:37.377590 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:37.383199 (+ 5609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:37.383204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:37.383229 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:37.383757 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.383766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.385209 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.385216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.388118 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.388132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.389371 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.389377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.389543 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.389590 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 18:45:37.389732 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 18:45:37.389942 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1911,"mutex_wait_us":279,"spinlock_wait_cycles":896,"thread_start_us":2832,"threads_started":1}
I20260620 18:45:37.390656 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.375317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.15.102.195:38721)
0620 18:45:37.382707 (+ 7390us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.382715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.382746 (+ 31us) server_negotiation.cc:407] Connection header received
0620 18:45:37.382836 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.382843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.383021 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.383206 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:37.383930 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.385076 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.388303 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.389188 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.389388 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.389553 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.390065 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.390279 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7222,"thread_start_us":7043,"threads_started":1}
I20260620 18:45:37.391059 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.378649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.15.102.194:40489)
0620 18:45:37.382339 (+ 3690us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.382351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.382379 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:45:37.382463 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.382470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.382720 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.382920 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:37.383958 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.385421 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.387982 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.389333 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.389504 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.389657 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.390587 (+ 930us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.390764 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3486,"thread_start_us":3343,"threads_started":1}
I20260620 18:45:37.391067 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.375787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.15.102.193:44597)
0620 18:45:37.381137 (+ 5350us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.381150 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.381186 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:45:37.381321 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.381334 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.381653 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.383373 (+ 1720us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:37.383626 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.385239 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.388267 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.389599 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.389793 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.389954 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.390573 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.390727 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5139,"thread_start_us":4944,"threads_started":1}
I20260620 18:45:37.391726 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.375116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44597 (local address 127.0.0.1:33234)
0620 18:45:37.379552 (+ 4436us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.379575 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:37.379715 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:37.382686 (+ 2971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:37.382695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:37.382723 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:37.383107 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.383127 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.385363 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.385368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.388118 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.388132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.389842 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.389848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.390139 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.390172 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:45:37.390269 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:45:37.390419 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4280,"mutex_wait_us":219,"thread_start_us":1718,"threads_started":1}
I20260620 18:45:37.392143 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.379417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:40489 (local address 127.0.0.1:56638)
0620 18:45:37.380332 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.380352 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:37.380465 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:37.383154 (+ 2689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:37.383160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:37.383190 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:37.383742 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.383757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.385544 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.385553 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.387822 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.387839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.389813 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.389822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.390134 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.390164 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:45:37.390253 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 18:45:37.390370 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":360,"thread_start_us":616,"threads_started":1}
I20260620 18:45:37.394630 26485 tablet_service.cc:1467] Processing CreateTablet for tablet 2d44680c5a754822ae2ead2b1e0a601a (DEFAULT_TABLE table=test-table [id=392c3b4351ce40849cdfe9690763be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:37.395184 26217 tablet_service.cc:1467] Processing CreateTablet for tablet 2d44680c5a754822ae2ead2b1e0a601a (DEFAULT_TABLE table=test-table [id=392c3b4351ce40849cdfe9690763be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:37.395184 26351 tablet_service.cc:1467] Processing CreateTablet for tablet 2d44680c5a754822ae2ead2b1e0a601a (DEFAULT_TABLE table=test-table [id=392c3b4351ce40849cdfe9690763be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:45:37.395857 26485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d44680c5a754822ae2ead2b1e0a601a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:37.396494 26351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d44680c5a754822ae2ead2b1e0a601a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:37.396494 26217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d44680c5a754822ae2ead2b1e0a601a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:45:37.407495 26569 tablet_bootstrap.cc:492] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: Bootstrap starting.
I20260620 18:45:37.410754 26570 tablet_bootstrap.cc:492] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Bootstrap starting.
I20260620 18:45:37.411055 26569 tablet_bootstrap.cc:654] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:37.411653 26571 tablet_bootstrap.cc:492] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: Bootstrap starting.
I20260620 18:45:37.412498 26569 log.cc:824] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:37.414592 26570 tablet_bootstrap.cc:654] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:37.414644 26571 tablet_bootstrap.cc:654] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: Neither blocks nor log segments found. Creating new log.
I20260620 18:45:37.415802 26571 log.cc:824] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:37.415854 26570 log.cc:824] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Log is configured to *not* fsync() on all Append() calls
I20260620 18:45:37.418105 26569 tablet_bootstrap.cc:492] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: No bootstrap required, opened a new log
I20260620 18:45:37.418397 26569 ts_tablet_manager.cc:1397] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260620 18:45:37.418787 26570 tablet_bootstrap.cc:492] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: No bootstrap required, opened a new log
I20260620 18:45:37.418895 26571 tablet_bootstrap.cc:492] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: No bootstrap required, opened a new log
I20260620 18:45:37.419128 26570 ts_tablet_manager.cc:1397] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 18:45:37.419147 26571 ts_tablet_manager.cc:1397] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 18:45:37.426292 26570 raft_consensus.cc:348] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.426785 26570 raft_consensus.cc:374] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:37.426923 26570 raft_consensus.cc:729] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458d3a53ac0c4e0d9ec10008ce3379b2, State: Initialized, Role: FOLLOWER
I20260620 18:45:37.427629 26570 consensus_queue.cc:260] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.428851 26416 heartbeater.cc:503] Master 127.15.102.254:35015 was elected leader, sending a full tablet report...
I20260620 18:45:37.429075 26569 raft_consensus.cc:348] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.429502 26569 raft_consensus.cc:374] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:37.429569 26570 ts_tablet_manager.cc:1428] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 18:45:37.429580 26569 raft_consensus.cc:729] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a81dd2ece1b844eb853d1303aec39757, State: Initialized, Role: FOLLOWER
I20260620 18:45:37.429438 26571 raft_consensus.cc:348] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.429821 26571 raft_consensus.cc:374] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:45:37.429925 26571 raft_consensus.cc:729] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30a795904f943f4a1317dad319ff07a, State: Initialized, Role: FOLLOWER
I20260620 18:45:37.430171 26569 consensus_queue.cc:260] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.430533 26571 consensus_queue.cc:260] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.431274 26550 heartbeater.cc:503] Master 127.15.102.254:35015 was elected leader, sending a full tablet report...
I20260620 18:45:37.432130 26569 ts_tablet_manager.cc:1428] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260620 18:45:37.436445 26571 ts_tablet_manager.cc:1428] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
I20260620 18:45:37.468962 26577 raft_consensus.cc:482] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:45:37.469276 26577 raft_consensus.cc:504] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.470911 26577 leader_election.cc:288] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a81dd2ece1b844eb853d1303aec39757 (127.15.102.195:38721), 458d3a53ac0c4e0d9ec10008ce3379b2 (127.15.102.194:40489)
W20260620 18:45:37.476035 26283 tablet.cc:2369] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:37.479049 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.471359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:46159 (local address 127.15.102.195:38721)
0620 18:45:37.471546 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.471550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.472020 (+ 470us) server_negotiation.cc:407] Connection header received
0620 18:45:37.472359 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.472369 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.472495 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.472574 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:37.473521 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.474397 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.476787 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.478047 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.478201 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.478248 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.478633 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.478775 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:45:37.479049 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.472218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45223 (local address 127.15.102.194:40489)
0620 18:45:37.472431 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.472437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.472960 (+ 523us) server_negotiation.cc:407] Connection header received
0620 18:45:37.473166 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.473171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.473273 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.473375 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:45:37.473938 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.474618 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.476619 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.477903 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.478058 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.478151 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.478633 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.478775 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:45:37.479197 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.471984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:40489 (local address 127.15.102.193:45223)
0620 18:45:37.472849 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.472863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:37.473033 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:37.473413 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:37.473418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:37.473441 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:37.473799 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.473810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.474768 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.474775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.476464 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.476476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.478126 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.478134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.478343 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.478393 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:45:37.478531 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:45:37.478690 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":5,"thread_start_us":591,"threads_started":1}
I20260620 18:45:37.479254 26578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.471202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:38721 (local address 127.15.102.193:46159)
0620 18:45:37.471917 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.471937 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:45:37.472148 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:45:37.472788 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:45:37.472792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:45:37.472818 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:45:37.473321 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.473331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.474540 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.474544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.476573 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:45:37.476587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.478240 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.478246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:45:37.478418 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.478463 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 18:45:37.478580 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:45:37.478710 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":50,"spinlock_wait_cycles":46336,"thread_start_us":345,"threads_started":1}
I20260620 18:45:37.480334 26371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d44680c5a754822ae2ead2b1e0a601a" candidate_uuid: "c30a795904f943f4a1317dad319ff07a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" is_pre_election: true
I20260620 18:45:37.480352 26505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d44680c5a754822ae2ead2b1e0a601a" candidate_uuid: "c30a795904f943f4a1317dad319ff07a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a81dd2ece1b844eb853d1303aec39757" is_pre_election: true
I20260620 18:45:37.480911 26371 raft_consensus.cc:2454] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30a795904f943f4a1317dad319ff07a in term 0.
I20260620 18:45:37.480911 26505 raft_consensus.cc:2454] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c30a795904f943f4a1317dad319ff07a in term 0.
I20260620 18:45:37.481571 26172 leader_election.cc:302] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a81dd2ece1b844eb853d1303aec39757, c30a795904f943f4a1317dad319ff07a; no voters:
I20260620 18:45:37.481937 26577 raft_consensus.cc:2781] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:45:37.482115 26577 raft_consensus.cc:482] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:45:37.482208 26577 raft_consensus.cc:3037] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:37.484540 26577 raft_consensus.cc:504] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.485172 26577 leader_election.cc:288] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [CANDIDATE]: Term 1 election: Requested vote from peers a81dd2ece1b844eb853d1303aec39757 (127.15.102.195:38721), 458d3a53ac0c4e0d9ec10008ce3379b2 (127.15.102.194:40489)
I20260620 18:45:37.485654 26505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d44680c5a754822ae2ead2b1e0a601a" candidate_uuid: "c30a795904f943f4a1317dad319ff07a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a81dd2ece1b844eb853d1303aec39757"
I20260620 18:45:37.485707 26371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d44680c5a754822ae2ead2b1e0a601a" candidate_uuid: "c30a795904f943f4a1317dad319ff07a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2"
I20260620 18:45:37.485879 26505 raft_consensus.cc:3037] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:37.485908 26371 raft_consensus.cc:3037] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:45:37.488123 26371 raft_consensus.cc:2454] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30a795904f943f4a1317dad319ff07a in term 1.
I20260620 18:45:37.488168 26505 raft_consensus.cc:2454] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c30a795904f943f4a1317dad319ff07a in term 1.
I20260620 18:45:37.488521 26172 leader_election.cc:302] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458d3a53ac0c4e0d9ec10008ce3379b2, c30a795904f943f4a1317dad319ff07a; no voters:
I20260620 18:45:37.488795 26577 raft_consensus.cc:2781] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:45:37.489216 26577 raft_consensus.cc:686] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 1 LEADER]: Becoming Leader. State: Replica: c30a795904f943f4a1317dad319ff07a, State: Running, Role: LEADER
I20260620 18:45:37.489768 26577 consensus_queue.cc:237] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } }
I20260620 18:45:37.494141 26093 catalog_manager.cc:5495] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a reported cstate change: term changed from 0 to 1, leader changed from <none> to c30a795904f943f4a1317dad319ff07a (127.15.102.193). New cstate: current_term: 1 leader_uuid: "c30a795904f943f4a1317dad319ff07a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30a795904f943f4a1317dad319ff07a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 } health_report { overall_health: UNKNOWN } } }
W20260620 18:45:37.505439 26417 tablet.cc:2369] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:45:37.510597 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.504659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.15.102.193:44597)
0620 18:45:37.504856 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.504862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.505023 (+ 161us) server_negotiation.cc:407] Connection header received
0620 18:45:37.505203 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.505211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.505365 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.505526 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:45:37.505864 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.506775 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.508103 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.508484 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.508574 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.509080 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:45:37.509571 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:45:37.509850 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.510150 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.510248 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 18:45:37.518469 26505 raft_consensus.cc:1261] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 1 FOLLOWER]: Refusing update from remote peer c30a795904f943f4a1317dad319ff07a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:37.518461 26371 raft_consensus.cc:1261] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 1 FOLLOWER]: Refusing update from remote peer c30a795904f943f4a1317dad319ff07a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:45:37.519459 26580 consensus_queue.cc:1035] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "458d3a53ac0c4e0d9ec10008ce3379b2" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 40489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:37.519852 26577 consensus_queue.cc:1035] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a81dd2ece1b844eb853d1303aec39757" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:45:37.546252 26092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:37.546546 26092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:45:37.550257 26092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d447525447a74471a10bdceec4b1eb9b: Sending DeleteTablet for 3 replicas of tablet 2d44680c5a754822ae2ead2b1e0a601a
I20260620 18:45:37.551517 26217 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d44680c5a754822ae2ead2b1e0a601a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:37 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:33234
I20260620 18:45:37.551553 26351 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d44680c5a754822ae2ead2b1e0a601a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:37 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:56638
I20260620 18:45:37.551559 26485 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d44680c5a754822ae2ead2b1e0a601a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:45:37 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:43800
I20260620 18:45:37.554358 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 26154
I20260620 18:45:37.564918 26595 tablet_replica.cc:330] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: stopping tablet replica
I20260620 18:45:37.566466 26595 raft_consensus.cc:2229] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:37.567322 26595 raft_consensus.cc:2258] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:37.570465 26593 tablet_replica.cc:330] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: stopping tablet replica
I20260620 18:45:37.571089 26594 tablet_replica.cc:330] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: stopping tablet replica
I20260620 18:45:37.572479 26594 raft_consensus.cc:2229] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:45:37.571762 26593 raft_consensus.cc:2229] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:45:37.572934 26595 ts_tablet_manager.cc:1905] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:45:37 dist-test-slave-zfp8 krb5kdc[26048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:45:37.573684 26593 raft_consensus.cc:2258] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:37.574148 26594 raft_consensus.cc:2258] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:45:37.578575 26593 ts_tablet_manager.cc:1905] T 2d44680c5a754822ae2ead2b1e0a601a P c30a795904f943f4a1317dad319ff07a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:37.579360 26594 ts_tablet_manager.cc:1905] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:45:37.580811 26595 ts_tablet_manager.cc:1918] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:37.581030 26595 log.cc:1196] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/2d44680c5a754822ae2ead2b1e0a601a
I20260620 18:45:37.581634 26595 ts_tablet_manager.cc:1939] T 2d44680c5a754822ae2ead2b1e0a601a P 458d3a53ac0c4e0d9ec10008ce3379b2: Deleting consensus metadata
I20260620 18:45:37.581984 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.555577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.15.102.193:44597)
0620 18:45:37.555750 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.555756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.561230 (+ 5474us) server_negotiation.cc:407] Connection header received
0620 18:45:37.561446 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.561455 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.561594 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.561734 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:37.563965 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.565284 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.567605 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.568458 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.568587 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.574310 (+ 5723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:37.574324 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:37.574342 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:37.574392 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:37.578375 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.579413 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.579423 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.579426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.579579 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.580096 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.580101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.580103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.580517 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:37.580797 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.581356 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.581628 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:45:37.583529 26277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:37.584030 26079 catalog_manager.cc:4841] TS 458d3a53ac0c4e0d9ec10008ce3379b2 (127.15.102.194:40489): tablet 2d44680c5a754822ae2ead2b1e0a601a (table test-table [id=392c3b4351ce40849cdfe9690763be98]) successfully deleted
I20260620 18:45:37.588907 26594 ts_tablet_manager.cc:1918] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:45:37.589093 26594 log.cc:1196] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/2d44680c5a754822ae2ead2b1e0a601a
I20260620 18:45:37.589839 26594 ts_tablet_manager.cc:1939] T 2d44680c5a754822ae2ead2b1e0a601a P a81dd2ece1b844eb853d1303aec39757: Deleting consensus metadata
I20260620 18:45:37.592252 26079 catalog_manager.cc:4841] TS a81dd2ece1b844eb853d1303aec39757 (127.15.102.195:38721): tablet 2d44680c5a754822ae2ead2b1e0a601a (table test-table [id=392c3b4351ce40849cdfe9690763be98]) successfully deleted
I20260620 18:45:37.684365 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 26154
W20260620 18:45:37.702587 26077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:44597 (error 108)
I20260620 18:45:37.702852 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 26288
I20260620 18:45:37.704718 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.704201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44597 (local address 127.0.0.1:33270)
0620 18:45:37.704388 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.704515 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:45:37.704955 26077 catalog_manager.cc:4584] TS c30a795904f943f4a1317dad319ff07a (127.15.102.193:44597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d44680c5a754822ae2ead2b1e0a601a: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44597: connect: Connection refused (error 111)
Jun 20 18:45:37 dist-test-slave-zfp8 krb5kdc[26048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:45:37.717799 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.703782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.15.102.194:40489)
0620 18:45:37.703955 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.703960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.703980 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:45:37.704171 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.704177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.704274 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.704381 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:37.705816 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.706584 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.707856 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.708245 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.708335 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.712338 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:37.712351 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:37.712367 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:37.712403 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:37.715089 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.715901 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.715909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.715912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.716043 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.716356 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.716360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.716362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.716777 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:37.716994 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.717248 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.717411 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:45:37.718540 26411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:37.759053 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.758489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44597 (local address 127.0.0.1:33272)
0620 18:45:37.758788 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.758897 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 18:45:37.760138 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.759824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44597 (local address 127.0.0.1:33282)
0620 18:45:37.759963 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.760039 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:45:37.808374 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 26288
I20260620 18:45:37.820995 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.820355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44597 (local address 127.0.0.1:33284)
0620 18:45:37.820635 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.820762 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:45:37.822158 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.821785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44597 (local address 127.0.0.1:33288)
0620 18:45:37.821957 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.822036 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:45:37.826058 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 26422
Jun 20 18:45:37 dist-test-slave-zfp8 krb5kdc[26048](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:45:37.841537 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.826865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.15.102.195:38721)
0620 18:45:37.827058 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:45:37.827065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:45:37.827089 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:45:37.827285 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:45:37.827292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:45:37.827359 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:45:37.827476 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:45:37.828907 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.829644 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.830924 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:45:37.831518 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:45:37.831621 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:45:37.836132 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:45:37.836148 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:45:37.836173 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:45:37.836214 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:45:37.838909 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.839660 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.839667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.839669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.839797 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:45:37.840123 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:45:37.840126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:45:37.840127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:45:37.840468 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:45:37.840732 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 18:45:37.840969 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:45:37.841135 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:45:37.842218 26545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:37.902714 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.902235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44597 (local address 127.0.0.1:33292)
0620 18:45:37.902470 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.902564 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 18:45:37.904096 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:45:37.903666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44597 (local address 127.0.0.1:33306)
0620 18:45:37.903906 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 18:45:37.903981 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 18:45:37.932736 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 26422
I20260620 18:45:37.950753 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 26061
I20260620 18:45:37.951973 26123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:45:38.059405 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 26061
2026-06-20T18:45:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4290 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:45:38 dist-test-slave-zfp8 krb5kdc[26604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:45:38 dist-test-slave-zfp8 krb5kdc[26604](info): set up 2 sockets
Jun 20 18:45:38 dist-test-slave-zfp8 krb5kdc[26604](info): commencing operation
krb5kdc: starting...
W20260620 18:45:39.164220 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:45:39 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:45:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.102.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.102.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/rangeradmin_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/rangeradmin_127.15.102.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.102.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.102.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/rangerlookup_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/rangerlookup_127.15.102.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.102.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.102.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/HTTP_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/HTTP_127.15.102.212@KRBTEST.COM.keytab.
I20260620 18:45:39.482352 15771 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskc6hSqA/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/postgres -l logfile start
2026-06-20 18:45:41.945 UTC [26634] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:45:41.946 UTC [26634] LOG: listening on IPv4 address "127.15.102.212", port 58419
2026-06-20 18:45:41.949 UTC [26634] LOG: listening on Unix socket "/tmp/.s.PGSQL.58419"
2026-06-20 18:45:41.971 UTC [26637] LOG: database system was shut down at 2026-06-20 18:45:40 UTC
2026-06-20 18:45:41.975 UTC [26634] LOG: database system is ready to accept connections
I20260620 18:45:42.967882 15771 mini_postgres.cc:96] Postgres bound to 58419
2026-06-20 18:45:42.985 UTC [26645] FATAL: database "slave" does not exist
127.15.102.212:58419 - accepting connections
I20260620 18:45:42.985934 15771 mini_ranger.cc:161] Starting Ranger...
I20260620 18:45:43.012224 15771 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:45:43.484712 15771 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:45:43.484861 15771 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-admin
2026-06-20 18:45:43,786 [I] DB FLAVOR :POSTGRES
2026-06-20 18:45:43,786 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:45:43,787 [I] Checking connection..
2026-06-20 18:45:43,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:45:44,047 [I] Checking connection passed.
2026-06-20 18:45:44,047 [I] --------- Verifying version history table ---------
2026-06-20 18:45:44,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:45:44,325 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:45:44,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:45:44,579 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:45:44,580 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:45:44,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:45:44.801 UTC [26709] WARNING: there is no transaction in progress
2026-06-20 18:45:44,807 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:45:44,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:45:45,065 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:45:45,065 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:45:45,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:45:45,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:45:45,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zfp8.c.gcp-upstream.internal','N') ;"
2026-06-20 18:45:45,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:45:46,080 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:45:46,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:45:46,311 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:45:46,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:45:46,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:45:46,797 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:45:46,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:45:47.766 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.782 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.799 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.814 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.830 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.847 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.909 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.916 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.930 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.940 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.950 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.962 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.972 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.978 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.985 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:47.992 UTC [26817] WARNING: there is no transaction in progress
2026-06-20 18:45:48,611 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:45:48,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:45:48,855 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:45:48,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:45:49,089 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:45:49,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:45:49,350 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:45:49,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:45:49,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:45:49,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zfp8.c.gcp-upstream.internal';"
2026-06-20 18:45:50,082 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:45:50,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:45:50,330 [I] DB_PATCHES have already been applied
I20260620 18:45:50.336741 15771 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-admin:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskc6hSqA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:45:50.336831 15771 mini_ranger.cc:193] Using host: 127.15.102.212
I20260620 18:45:50.351661 15771 mini_ranger.cc:227] Ranger admin URL: http://127.15.102.212:46797
Jun 20, 2026 6:45:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:45:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:45:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:45:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:45:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:45:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:45:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:45:51.502832 15771 mini_ranger.cc:160] Time spent starting Ranger: real 8.517s user 0.000s sys 0.040s
Jun 20, 2026 6:46:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:46:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:46:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:46:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:46:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:46:13.456033 15771 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.102.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.102.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/rangerkms_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/rangerkms_127.15.102.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.102.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.102.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.102.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/HTTP_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.102.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/HTTP_127.15.102.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:46:13 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981173, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:46:13.618877 15771 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:46:13.640327 15771 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:46:14.546833 15771 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:46:14.547014 15771 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms
I20260620 18:46:14.547142 15771 mini_ranger_kms.cc:227] Using postgres at 127.15.102.212:58419
2026-06-20 18:46:14,961 [I] DB FLAVOR :POSTGRES
2026-06-20 18:46:14,961 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:46:14,961 [I] Checking connection
2026-06-20 18:46:14,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:46:15,189 [I] connection success
2026-06-20 18:46:15,190 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:46:15,190 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:46:15,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:46:15,423 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:46:15,423 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:46:15,423 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:46:15,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:58419/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:46:15,676 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:46:16.119158 15771 mini_ranger_kms.cc:310] Created kms service
I20260620 18:46:16.326896 15771 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:46:16.371557 15771 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:46:16 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981173, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:46:16.771510 15771 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:46:16.771833 15771 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms:/tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:46:16.771891 15771 mini_ranger_kms.cc:240] Using host: 127.15.102.212
I20260620 18:46:16.792220 15771 mini_ranger_kms.cc:276] Ranger KMS PID: 26997
I20260620 18:46:16.792426 15771 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.102.212:49493
Jun 20, 2026 6:46:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:46:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:46:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:46:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:46:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:46:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 18:46:19 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981179, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.102.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:19.092049 15771 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.473s user 0.113s sys 0.328s
I20260620 18:46:19.092314 15771 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:46:19.092531 15771 mini_ranger_kms.cc:401] 127.15.102.212:49493/kms/v1/keys
Jun 20 18:46:19 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981179, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.102.212@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
Jun 20 18:46:19 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981179, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.102.212@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
Jun 20 18:46:22 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:46:22.384953 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:39649
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:39649
--ranger_config_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:49493/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:46:22.771201 27048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:22.771679 27048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:22.771848 27048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:22.779559 27048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:46:22.779729 27048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:22.779847 27048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:22.779951 27048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:46:22.780040 27048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:46:22.791155 27048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.102.212:49493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:39649
--ranger_config_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:39649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:46:22.792732 27048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:22.794288 27048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:22.807543 27054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:22.807559 27056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:22.808399 27053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:22.808852 27048 server_base.cc:1034] running on GCE node
I20260620 18:46:22.809772 27048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:22.811503 27048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:22.812742 27048 hybrid_clock.cc:648] HybridClock initialized: now 1781981182812706 us; error 39 us; skew 500 ppm
I20260620 18:46:22.813153 27048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:22 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981182, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:22.816536 27048 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:46:22.818194 27048 webserver.cc:460] Webserver started at http://127.15.102.254:39323/ using document root <none> and password file <none>
I20260620 18:46:22.818912 27048 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:22.819054 27048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:22.819437 27048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:46:22 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981182, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:46:22.885844 27048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "8be71ce6fbd64ae298b06e149df04702"
format_stamp: "Formatted at 2026-06-20 18:46:22 on dist-test-slave-zfp8"
server_key: "0478f5161123f0d9feab107caab08c4b"
server_key_iv: "c4aa5754eb09470ca7455a4c4463218a"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:22.886773 27048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "8be71ce6fbd64ae298b06e149df04702"
format_stamp: "Formatted at 2026-06-20 18:46:22 on dist-test-slave-zfp8"
server_key: "0478f5161123f0d9feab107caab08c4b"
server_key_iv: "c4aa5754eb09470ca7455a4c4463218a"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:22.892890 27048 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 18:46:22.915432 27063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:22.917188 27048 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:46:22.917397 27048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "8be71ce6fbd64ae298b06e149df04702"
format_stamp: "Formatted at 2026-06-20 18:46:22 on dist-test-slave-zfp8"
server_key: "0478f5161123f0d9feab107caab08c4b"
server_key_iv: "c4aa5754eb09470ca7455a4c4463218a"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:22.917654 27048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:22.936113 27048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:22.941435 27048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:22.941789 27048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:22.964915 27048 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:39649
I20260620 18:46:22.964936 27114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:39649 every 8 connection(s)
I20260620 18:46:22.966895 27048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:46:22.970002 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 27048
I20260620 18:46:22.970220 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:46:22.970992 15771 external_mini_cluster.cc:1428] Setting key 2e52df3c3b09daf3d4813a56809aa661
I20260620 18:46:22.971942 27115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:46:22 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:46:22.988813 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702: Bootstrap starting.
I20260620 18:46:22.992871 27115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:22.994350 27115 log.cc:824] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:22.998245 27115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702: No bootstrap required, opened a new log
I20260620 18:46:22.998303 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:22.974307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.15.102.254:39649)
0620 18:46:22.975176 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:22.975194 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:22.975287 (+ 93us) server_negotiation.cc:407] Connection header received
0620 18:46:22.977499 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:22.977536 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:22.978186 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:22.978749 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:22.980077 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:22.982107 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:22.983682 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:22.984935 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:22.985197 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:22.989781 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:22.989790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:22.989808 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:22.989857 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:22.993212 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:22.994206 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:22.994213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:22.994220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:22.994417 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:22.994914 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:22.994917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:22.994920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:22.995599 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:22.995839 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:22.996426 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 18:46:22.996861 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":362,"threads_started":1}
I20260620 18:46:23.007514 27115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be71ce6fbd64ae298b06e149df04702" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39649 } }
I20260620 18:46:23.008008 27115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:23.008090 27115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be71ce6fbd64ae298b06e149df04702, State: Initialized, Role: FOLLOWER
I20260620 18:46:23.008721 27115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be71ce6fbd64ae298b06e149df04702" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39649 } }
I20260620 18:46:23.009032 27115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:46:23.009152 27115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:46:23.009280 27115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:23.012060 27115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be71ce6fbd64ae298b06e149df04702" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39649 } }
I20260620 18:46:23.012598 27115 leader_election.cc:302] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8be71ce6fbd64ae298b06e149df04702; no voters:
I20260620 18:46:23.013190 27115 leader_election.cc:288] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:46:23.013386 27120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:23.014076 27120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [term 1 LEADER]: Becoming Leader. State: Replica: 8be71ce6fbd64ae298b06e149df04702, State: Running, Role: LEADER
I20260620 18:46:23.014710 27120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be71ce6fbd64ae298b06e149df04702" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39649 } }
I20260620 18:46:23.015571 27115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:46:23.017867 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8be71ce6fbd64ae298b06e149df04702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be71ce6fbd64ae298b06e149df04702" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39649 } } }
I20260620 18:46:23.018399 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:23.017756 27122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8be71ce6fbd64ae298b06e149df04702. Latest consensus state: current_term: 1 leader_uuid: "8be71ce6fbd64ae298b06e149df04702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be71ce6fbd64ae298b06e149df04702" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39649 } } }
I20260620 18:46:23.018695 27122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702 [sys.catalog]: This master's current role is: LEADER
I20260620 18:46:23.027777 27126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:46:23.034440 27126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:46:23.035382 27115 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:46:23.053586 27126 catalog_manager.cc:1347] Generated new cluster ID: 35e97020c93241cdb82a8c879eb0d1e0
I20260620 18:46:23.053752 27126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:46:23.081218 27126 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:46:23.082825 27126 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:46:23.097345 27126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702: Generated new TSK 0
I20260620 18:46:23.098356 27126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:46:23.917611 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:49493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:39649
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:46:24 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981184, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:46:24.378049 27160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:24.378408 27160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:24.378587 27160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:24.386864 27160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:24.387063 27160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:24.387333 27160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:46:24.401190 27160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.102.212:49493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:46:24.402853 27160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:24.404527 27160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:24.431951 27170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:24.433074 27171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:24.445223 27173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:24.455269 27160 server_base.cc:1034] running on GCE node
I20260620 18:46:24.456355 27160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:24.458763 27160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:24.461236 27160 hybrid_clock.cc:648] HybridClock initialized: now 1781981184461035 us; error 193 us; skew 500 ppm
I20260620 18:46:24.461737 27160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:24 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981184, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:24.466825 27160 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:46:24.475782 27160 webserver.cc:460] Webserver started at http://127.15.102.193:33425/ using document root <none> and password file <none>
I20260620 18:46:24.476936 27160 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:24.477110 27160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:24.477552 27160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:46:24 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981184, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:46:24.503181 27160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "d9bb072bd2c5415cb47b20ec2e713621"
format_stamp: "Formatted at 2026-06-20 18:46:24 on dist-test-slave-zfp8"
server_key: "9c656fe5f2ddf7e6699ea87b155d839b"
server_key_iv: "243b02f8bc23fdf9323fae5e98a392b0"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:24.504529 27160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "d9bb072bd2c5415cb47b20ec2e713621"
format_stamp: "Formatted at 2026-06-20 18:46:24 on dist-test-slave-zfp8"
server_key: "9c656fe5f2ddf7e6699ea87b155d839b"
server_key_iv: "243b02f8bc23fdf9323fae5e98a392b0"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:24.516414 27160 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.007s
I20260620 18:46:24.536612 27180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:24.539217 27160 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260620 18:46:24.539497 27160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "d9bb072bd2c5415cb47b20ec2e713621"
format_stamp: "Formatted at 2026-06-20 18:46:24 on dist-test-slave-zfp8"
server_key: "9c656fe5f2ddf7e6699ea87b155d839b"
server_key_iv: "243b02f8bc23fdf9323fae5e98a392b0"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:24.539800 27160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:24.584362 27160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:24.590346 27160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:24.590785 27160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:24.602402 27160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:24.610989 27160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:24.611207 27160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:46:24.611449 27160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:24.611569 27160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:24.738391 27160 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:38087
I20260620 18:46:24.738830 27292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:38087 every 8 connection(s)
I20260620 18:46:24.740372 27160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:46:24.747040 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 27160
I20260620 18:46:24.747287 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:46:24.748129 15771 external_mini_cluster.cc:1428] Setting key b64f45cfd8f7ddcc43b482513f77a9b1
Jun 20 18:46:24 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981184, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:46:24.786510 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:24.744887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:55797 (local address 127.15.102.254:39649)
0620 18:46:24.764243 (+ 19356us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:24.764251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:24.764276 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:46:24.764349 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:24.764355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:24.764447 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:24.764571 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:24.767973 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:24.769169 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:24.772418 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:24.773113 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:24.773239 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:24.778393 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:24.778401 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:24.778406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:24.778456 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:24.782197 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:24.783420 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:24.783428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:24.783431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:24.783593 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:24.784234 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:24.784239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:24.784243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:24.784611 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:24.784750 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:24.785843 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0620 18:46:24.786092 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18442,"thread_start_us":18264,"threads_started":1}
Jun 20 18:46:24 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981184, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:46:24.789870 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:24.747942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:39649 (local address 127.15.102.193:55797)
0620 18:46:24.758587 (+ 10645us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:24.758673 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:24.761326 (+ 2653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:24.765956 (+ 4630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:24.765972 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:24.766687 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:24.767683 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:24.767719 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:24.769419 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:24.769425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:24.772185 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:24.772206 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:24.773630 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:24.773638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:24.773977 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:24.774903 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:24.774944 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:24.777993 (+ 3049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:24.782494 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:24.782506 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:24.782528 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:24.783254 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:24.783763 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:24.783768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:24.783771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:24.784080 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:24.784853 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:24.784862 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:24.785381 (+ 519us) client_negotiation.cc:769] Sending connection context
0620 18:46:24.787037 (+ 1656us) client_negotiation.cc:240] Negotiation successful
0620 18:46:24.787693 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10207,"thread_start_us":13815,"threads_started":1}
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:46:24.792145 27293 heartbeater.cc:347] Connected to a master server at 127.15.102.254:39649
I20260620 18:46:24.792502 27293 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:24.793452 27293 heartbeater.cc:511] Master 127.15.102.254:39649 requested a full tablet report, sending...
I20260620 18:46:24.796048 27080 ts_manager.cc:194] Registered new tserver with Master: d9bb072bd2c5415cb47b20ec2e713621 (127.15.102.193:38087)
I20260620 18:46:24.798225 27080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:55797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:46:24.848732 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:49493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:39649
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:46:25 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981184, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
W20260620 18:46:25.357344 27299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:25.357645 27299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:25.357810 27299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:25.365777 27299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:25.365908 27299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:25.366087 27299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:46:25.376461 27299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.102.212:49493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:46:25.377941 27299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:25.379418 27299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:25.394341 27313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:25.394341 27314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:25.394388 27316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:25.394829 27299 server_base.cc:1034] running on GCE node
I20260620 18:46:25.395747 27299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:25.396795 27299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:25.398022 27299 hybrid_clock.cc:648] HybridClock initialized: now 1781981185397973 us; error 44 us; skew 500 ppm
I20260620 18:46:25.398406 27299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:25 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981185, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:25.401947 27299 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:46:25.403712 27299 webserver.cc:460] Webserver started at http://127.15.102.194:42645/ using document root <none> and password file <none>
I20260620 18:46:25.404593 27299 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:25.404754 27299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:25.405097 27299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:46:25 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981185, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:46:25.419570 27299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "0cd88091f8c5425980986ee1f14e1d5f"
format_stamp: "Formatted at 2026-06-20 18:46:25 on dist-test-slave-zfp8"
server_key: "5ad090ab1db745d40bf0351d0d64a12e"
server_key_iv: "433211b94efcbc5d259a264269adc3ba"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:25.420502 27299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "0cd88091f8c5425980986ee1f14e1d5f"
format_stamp: "Formatted at 2026-06-20 18:46:25 on dist-test-slave-zfp8"
server_key: "5ad090ab1db745d40bf0351d0d64a12e"
server_key_iv: "433211b94efcbc5d259a264269adc3ba"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:25.427039 27299 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 18:46:25.442584 27323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:25.444787 27299 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:46:25.445021 27299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "0cd88091f8c5425980986ee1f14e1d5f"
format_stamp: "Formatted at 2026-06-20 18:46:25 on dist-test-slave-zfp8"
server_key: "5ad090ab1db745d40bf0351d0d64a12e"
server_key_iv: "433211b94efcbc5d259a264269adc3ba"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:25.445385 27299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:25.463830 27299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:25.468968 27299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:25.469321 27299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:25.470363 27299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:25.471894 27299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:25.471995 27299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:25.472146 27299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:25.472215 27299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:25.512733 27299 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:36245
I20260620 18:46:25.512791 27435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:36245 every 8 connection(s)
I20260620 18:46:25.514175 27299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:46:25.516474 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 27299
I20260620 18:46:25.516700 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:46:25.517328 15771 external_mini_cluster.cc:1428] Setting key 70faba81379d6ffe21da1f37274e8b04
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Jun 20 18:46:25 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981185, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
I20260620 18:46:25.554697 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:25.517403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:46371 (local address 127.15.102.254:39649)
0620 18:46:25.527700 (+ 10297us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:25.527706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:25.532656 (+ 4950us) server_negotiation.cc:407] Connection header received
0620 18:46:25.535651 (+ 2995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:25.535661 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:25.535771 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:25.535911 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:25.538210 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:25.539504 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:25.542165 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:25.542910 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:25.543087 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:25.547809 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:25.547818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:25.547823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:25.547870 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:25.551333 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:25.552188 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:25.552194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:25.552201 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:25.552357 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:25.552832 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:25.552837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:25.552841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:25.553137 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:25.553301 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:25.554050 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 18:46:25.554311 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10176,"thread_start_us":10041,"threads_started":1}
I20260620 18:46:25.555948 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:25.518136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:39649 (local address 127.15.102.194:46371)
0620 18:46:25.532429 (+ 14293us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:25.532500 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:25.535183 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:25.536263 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:25.536277 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:25.536971 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:25.537950 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:25.537981 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:25.539724 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:25.539730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:25.541896 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:25.541918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:25.543220 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:25.543229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:25.543617 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:25.544505 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:25.544538 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:25.547489 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:25.551512 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:25.551526 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:25.551544 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:25.552042 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:25.552475 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:25.552480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:25.552484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:25.552689 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:25.553270 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:25.553280 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:25.553657 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 18:46:25.553956 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 18:46:25.554530 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13805,"thread_start_us":13537,"threads_started":1}
I20260620 18:46:25.557633 27436 heartbeater.cc:347] Connected to a master server at 127.15.102.254:39649
I20260620 18:46:25.557992 27436 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:25.559005 27436 heartbeater.cc:511] Master 127.15.102.254:39649 requested a full tablet report, sending...
I20260620 18:46:25.560845 27080 ts_manager.cc:194] Registered new tserver with Master: 0cd88091f8c5425980986ee1f14e1d5f (127.15.102.194:36245)
I20260620 18:46:25.561736 27080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:46371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:46:25.586305 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:49493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:39649
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:46:25.802711 27293 heartbeater.cc:503] Master 127.15.102.254:39649 was elected leader, sending a full tablet report...
W20260620 18:46:25.977807 27442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:46:25.978106 27442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:46:25.978273 27442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:46:25.985759 27442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:46:25.985880 27442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:46:25.986069 27442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:46:25.996013 27442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.102.212:49493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:39649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:46:25.997469 27442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:46:25.998793 27442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:46:26.012975 27447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:26.013413 27450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:46:26.013386 27448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:46:26.014529 27442 server_base.cc:1034] running on GCE node
I20260620 18:46:26.015167 27442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:46:26.016253 27442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:46:26.017506 27442 hybrid_clock.cc:648] HybridClock initialized: now 1781981186017442 us; error 53 us; skew 500 ppm
I20260620 18:46:26.017870 27442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:46:26 dist-test-slave-zfp8 krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981186, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:46:26.023960 27442 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:46:26.025682 27442 webserver.cc:460] Webserver started at http://127.15.102.195:33033/ using document root <none> and password file <none>
I20260620 18:46:26.026516 27442 fs_manager.cc:359] Metadata directory not provided
I20260620 18:46:26.026674 27442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:46:26.027014 27442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:46:26 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981186, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:46:26.043545 27442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "6b029536301247338d41b9c4278d9656"
format_stamp: "Formatted at 2026-06-20 18:46:26 on dist-test-slave-zfp8"
server_key: "e1db36cd63c7ea210dba0a5960620f06"
server_key_iv: "f710b13fcf5bf5f1c1f5648cf3a95bd9"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:26.044515 27442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "6b029536301247338d41b9c4278d9656"
format_stamp: "Formatted at 2026-06-20 18:46:26 on dist-test-slave-zfp8"
server_key: "e1db36cd63c7ea210dba0a5960620f06"
server_key_iv: "f710b13fcf5bf5f1c1f5648cf3a95bd9"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:26.051448 27442 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 18:46:26.068852 27457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:26.070587 27442 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:46:26.070832 27442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "6b029536301247338d41b9c4278d9656"
format_stamp: "Formatted at 2026-06-20 18:46:26 on dist-test-slave-zfp8"
server_key: "e1db36cd63c7ea210dba0a5960620f06"
server_key_iv: "f710b13fcf5bf5f1c1f5648cf3a95bd9"
server_key_version: "kuduclusterkey@0"
I20260620 18:46:26.071081 27442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:46:26.097462 27442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:46:26.101426 27442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:46:26.101760 27442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:46:26.102737 27442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:46:26.104575 27442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:46:26.104686 27442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:26.104835 27442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:46:26.104907 27442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:46:26.141754 27442 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:34631
I20260620 18:46:26.141841 27569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:34631 every 8 connection(s)
I20260620 18:46:26.143062 27442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:46:26.145354 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 27442
I20260620 18:46:26.145587 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:46:26.146149 15771 external_mini_cluster.cc:1428] Setting key cbf11ce749edc00b279020734a48252c
Jun 20 18:46:26 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981186, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:46:26.168766 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.145974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:60751 (local address 127.15.102.254:39649)
0620 18:46:26.154617 (+ 8643us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.154621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.154636 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:46:26.154685 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.154688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.154753 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.154876 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:26.156481 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.157204 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.158828 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.159324 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.159471 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.163205 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:26.163210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:26.163214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:26.163297 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:26.165847 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.166626 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.166630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.166632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.166758 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.167128 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.167131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.167133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.167491 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:26.167615 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.168310 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.168502 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8535,"thread_start_us":8382,"threads_started":1}
I20260620 18:46:26.169866 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.146890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:39649 (local address 127.15.102.195:60751)
0620 18:46:26.151814 (+ 4924us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.151869 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:26.154004 (+ 2135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:26.155150 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:26.155160 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:26.155681 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:46:26.156308 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.156327 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.157382 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.157386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.158648 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.158666 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.159591 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.159598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.159847 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.160707 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:46:26.160738 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:46:26.162993 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:46:26.166012 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:26.166020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:26.166035 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:26.166522 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:26.166851 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:46:26.166854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:46:26.166856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:46:26.167040 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:46:26.167613 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:46:26.167619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:46:26.167980 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 18:46:26.168245 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 18:46:26.168659 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4623,"thread_start_us":4354,"threads_started":1}
I20260620 18:46:26.171314 27570 heartbeater.cc:347] Connected to a master server at 127.15.102.254:39649
I20260620 18:46:26.171630 27570 heartbeater.cc:464] Registering TS with master...
I20260620 18:46:26.172502 27570 heartbeater.cc:511] Master 127.15.102.254:39649 requested a full tablet report, sending...
I20260620 18:46:26.174090 27080 ts_manager.cc:194] Registered new tserver with Master: 6b029536301247338d41b9c4278d9656 (127.15.102.195:34631)
I20260620 18:46:26.175025 27080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:60751
I20260620 18:46:26.187305 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:46:26.207046 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.193096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.15.102.254:39649)
0620 18:46:26.193284 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.193289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.193768 (+ 479us) server_negotiation.cc:407] Connection header received
0620 18:46:26.193925 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.193930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.194011 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.194107 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:26.195541 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.196291 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.197732 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.198166 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.198267 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.200930 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:26.200936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:26.200940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:26.200981 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:26.204197 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.205017 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.205023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.205027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.205174 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.205568 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.205575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.205578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.205875 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:26.206138 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.206468 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.206709 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:46:26.211377 27080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:46:26.219307 27080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:46:26.257555 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.234867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.15.102.194:36245)
0620 18:46:26.241745 (+ 6878us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.241761 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.245204 (+ 3443us) server_negotiation.cc:407] Connection header received
0620 18:46:26.246903 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.246917 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.247358 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.247641 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:26.249458 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.252002 (+ 2544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.254923 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.256220 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.256412 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.256611 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.256974 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.257166 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6591,"thread_start_us":6384,"threads_started":1}
I20260620 18:46:26.258031 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.233809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:36245 (local address 127.0.0.1:53552)
0620 18:46:26.244845 (+ 11036us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.244875 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:26.246578 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:26.248044 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:26.248057 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:26.248107 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:26.249138 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.249165 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.252167 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.252173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.254697 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.254717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.256459 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.256466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.256666 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.256735 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 18:46:26.256906 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 18:46:26.257124 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10824,"mutex_wait_us":2,"thread_start_us":14277,"threads_started":1}
I20260620 18:46:26.258492 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.233481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:34631 (local address 127.0.0.1:46148)
0620 18:46:26.234589 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.234636 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:26.234873 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:26.251325 (+ 16452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:26.251330 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:26.251364 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:26.251744 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.251783 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.253550 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.253556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.255663 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.255679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.257700 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.257714 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.257891 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.257932 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:46:26.258014 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 18:46:26.258160 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":724,"threads_started":1}
I20260620 18:46:26.259639 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.233819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.15.102.195:34631)
0620 18:46:26.250641 (+ 16822us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.250651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.250691 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:46:26.250836 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.250842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.251068 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.251343 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:26.252027 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.253379 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.255866 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.257250 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.258790 (+ 1540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.258990 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.259072 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.259280 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16640,"thread_start_us":16417,"threads_started":1}
I20260620 18:46:26.260548 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.235327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:38087 (local address 127.0.0.1:45308)
0620 18:46:26.247752 (+ 12425us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.247784 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:26.247967 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:26.251819 (+ 3852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:26.251829 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:26.251860 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:26.252309 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.252321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.254948 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.254954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.256832 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.256859 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.259777 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.259784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.259966 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.260001 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:46:26.260068 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 18:46:26.260175 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12224,"thread_start_us":9932,"threads_started":1}
I20260620 18:46:26.261554 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.235429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.15.102.193:38087)
0620 18:46:26.250913 (+ 15484us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.250922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.250951 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:46:26.251048 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.251055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.251342 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.251807 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:26.252501 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.254765 (+ 2264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.257032 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.259490 (+ 2458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.260860 (+ 1370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.261035 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.261110 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.261286 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15283,"thread_start_us":16080,"threads_started":1}
I20260620 18:46:26.262764 27371 tablet_service.cc:1467] Processing CreateTablet for tablet e044e0383a344f858467408ae82549cd (DEFAULT_TABLE table=test-table [id=b02b5726eefa4c8f9a589bd7efec516e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:26.264441 27371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e044e0383a344f858467408ae82549cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:26.264243 27505 tablet_service.cc:1467] Processing CreateTablet for tablet e044e0383a344f858467408ae82549cd (DEFAULT_TABLE table=test-table [id=b02b5726eefa4c8f9a589bd7efec516e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:26.264598 27228 tablet_service.cc:1467] Processing CreateTablet for tablet e044e0383a344f858467408ae82549cd (DEFAULT_TABLE table=test-table [id=b02b5726eefa4c8f9a589bd7efec516e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:46:26.266022 27228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e044e0383a344f858467408ae82549cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:26.266224 27505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e044e0383a344f858467408ae82549cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:46:26.279843 27589 tablet_bootstrap.cc:492] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: Bootstrap starting.
I20260620 18:46:26.283721 27589 tablet_bootstrap.cc:654] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:26.284049 27591 tablet_bootstrap.cc:492] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Bootstrap starting.
I20260620 18:46:26.285362 27589 log.cc:824] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:26.286782 27590 tablet_bootstrap.cc:492] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: Bootstrap starting.
I20260620 18:46:26.288229 27591 tablet_bootstrap.cc:654] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:26.289220 27589 tablet_bootstrap.cc:492] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: No bootstrap required, opened a new log
I20260620 18:46:26.289713 27589 ts_tablet_manager.cc:1397] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 18:46:26.289848 27591 log.cc:824] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:26.291033 27590 tablet_bootstrap.cc:654] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: Neither blocks nor log segments found. Creating new log.
I20260620 18:46:26.292567 27590 log.cc:824] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:46:26.293567 27591 tablet_bootstrap.cc:492] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: No bootstrap required, opened a new log
I20260620 18:46:26.293865 27591 ts_tablet_manager.cc:1397] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 18:46:26.296591 27590 tablet_bootstrap.cc:492] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: No bootstrap required, opened a new log
I20260620 18:46:26.296960 27590 ts_tablet_manager.cc:1397] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:46:26.298976 27589 raft_consensus.cc:348] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.299523 27589 raft_consensus.cc:374] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:26.299707 27589 raft_consensus.cc:729] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9bb072bd2c5415cb47b20ec2e713621, State: Initialized, Role: FOLLOWER
I20260620 18:46:26.300696 27589 consensus_queue.cc:260] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.308013 27590 raft_consensus.cc:348] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.308600 27590 raft_consensus.cc:374] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:26.308732 27590 raft_consensus.cc:729] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd88091f8c5425980986ee1f14e1d5f, State: Initialized, Role: FOLLOWER
I20260620 18:46:26.308995 27591 raft_consensus.cc:348] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.309705 27591 raft_consensus.cc:374] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:46:26.309542 27590 consensus_queue.cc:260] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.309916 27591 raft_consensus.cc:729] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b029536301247338d41b9c4278d9656, State: Initialized, Role: FOLLOWER
I20260620 18:46:26.310732 27591 consensus_queue.cc:260] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.319484 27589 ts_tablet_manager.cc:1428] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: Time spent starting tablet: real 0.030s user 0.018s sys 0.011s
I20260620 18:46:26.319765 27590 ts_tablet_manager.cc:1428] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: Time spent starting tablet: real 0.023s user 0.014s sys 0.007s
I20260620 18:46:26.320307 27436 heartbeater.cc:503] Master 127.15.102.254:39649 was elected leader, sending a full tablet report...
I20260620 18:46:26.320569 27570 heartbeater.cc:503] Master 127.15.102.254:39649 was elected leader, sending a full tablet report...
I20260620 18:46:26.321305 27591 ts_tablet_manager.cc:1428] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Time spent starting tablet: real 0.027s user 0.018s sys 0.009s
I20260620 18:46:26.324708 27596 raft_consensus.cc:482] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:46:26.325067 27596 raft_consensus.cc:504] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.327173 27596 leader_election.cc:288] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b029536301247338d41b9c4278d9656 (127.15.102.195:34631), d9bb072bd2c5415cb47b20ec2e713621 (127.15.102.193:38087)
I20260620 18:46:26.335793 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.327523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:45431 (local address 127.15.102.195:34631)
0620 18:46:26.327720 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.327724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.328275 (+ 551us) server_negotiation.cc:407] Connection header received
0620 18:46:26.328507 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.328514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.328618 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.328720 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:26.329882 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.330834 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.333183 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.334474 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.334687 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.334773 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.335379 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.335519 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 18:46:26.335793 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.328635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:42483 (local address 127.15.102.193:38087)
0620 18:46:26.328829 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.328834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.329218 (+ 384us) server_negotiation.cc:407] Connection header received
0620 18:46:26.329427 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.329434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.329531 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.329634 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:46:26.330370 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.331284 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.333311 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.334538 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.334692 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.334767 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.335420 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.335549 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:46:26.336036 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.327481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:34631 (local address 127.15.102.194:45431)
0620 18:46:26.328152 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.328178 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:26.328321 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:26.328935 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:26.328939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:26.328969 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:26.329720 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.329730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.330979 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.330985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.333026 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.333039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.334852 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.334858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.335167 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.335215 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:46:26.335341 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 18:46:26.335528 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":152,"thread_start_us":341,"threads_started":1}
I20260620 18:46:26.336066 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.328321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:38087 (local address 127.15.102.194:42483)
0620 18:46:26.329101 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 18:46:26.329120 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:46:26.329277 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:46:26.329655 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:46:26.329660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:46:26.329693 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:46:26.330157 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.330171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.331435 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.331439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.333083 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:46:26.333097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.334851 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.334858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:46:26.335153 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.335212 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:46:26.335341 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 18:46:26.335528 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":51,"thread_start_us":290,"threads_started":1}
I20260620 18:46:26.337249 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e044e0383a344f858467408ae82549cd" candidate_uuid: "0cd88091f8c5425980986ee1f14e1d5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b029536301247338d41b9c4278d9656" is_pre_election: true
I20260620 18:46:26.337237 27248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e044e0383a344f858467408ae82549cd" candidate_uuid: "0cd88091f8c5425980986ee1f14e1d5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9bb072bd2c5415cb47b20ec2e713621" is_pre_election: true
I20260620 18:46:26.337778 27248 raft_consensus.cc:2454] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cd88091f8c5425980986ee1f14e1d5f in term 0.
I20260620 18:46:26.337780 27525 raft_consensus.cc:2454] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cd88091f8c5425980986ee1f14e1d5f in term 0.
I20260620 18:46:26.338461 27324 leader_election.cc:302] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cd88091f8c5425980986ee1f14e1d5f, 6b029536301247338d41b9c4278d9656; no voters:
I20260620 18:46:26.338912 27596 raft_consensus.cc:2781] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:46:26.339092 27596 raft_consensus.cc:482] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:46:26.339226 27596 raft_consensus.cc:3037] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:26.341768 27596 raft_consensus.cc:504] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.342519 27596 leader_election.cc:288] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [CANDIDATE]: Term 1 election: Requested vote from peers 6b029536301247338d41b9c4278d9656 (127.15.102.195:34631), d9bb072bd2c5415cb47b20ec2e713621 (127.15.102.193:38087)
I20260620 18:46:26.343041 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e044e0383a344f858467408ae82549cd" candidate_uuid: "0cd88091f8c5425980986ee1f14e1d5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b029536301247338d41b9c4278d9656"
I20260620 18:46:26.343102 27248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e044e0383a344f858467408ae82549cd" candidate_uuid: "0cd88091f8c5425980986ee1f14e1d5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9bb072bd2c5415cb47b20ec2e713621"
I20260620 18:46:26.343441 27248 raft_consensus.cc:3037] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:26.343730 27525 raft_consensus.cc:3037] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:46:26.346138 27525 raft_consensus.cc:2454] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cd88091f8c5425980986ee1f14e1d5f in term 1.
I20260620 18:46:26.346216 27248 raft_consensus.cc:2454] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cd88091f8c5425980986ee1f14e1d5f in term 1.
I20260620 18:46:26.346783 27324 leader_election.cc:302] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cd88091f8c5425980986ee1f14e1d5f, 6b029536301247338d41b9c4278d9656; no voters:
I20260620 18:46:26.347175 27596 raft_consensus.cc:2781] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:46:26.347725 27596 raft_consensus.cc:686] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 1 LEADER]: Becoming Leader. State: Replica: 0cd88091f8c5425980986ee1f14e1d5f, State: Running, Role: LEADER
I20260620 18:46:26.348290 27596 consensus_queue.cc:237] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } }
I20260620 18:46:26.354094 27080 catalog_manager.cc:5495] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cd88091f8c5425980986ee1f14e1d5f (127.15.102.194). New cstate: current_term: 1 leader_uuid: "0cd88091f8c5425980986ee1f14e1d5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cd88091f8c5425980986ee1f14e1d5f" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 } health_report { overall_health: UNKNOWN } } }
I20260620 18:46:26.390741 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.385225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.15.102.194:36245)
0620 18:46:26.385475 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.385483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.385518 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:46:26.385620 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.385624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.385717 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.385875 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:46:26.386236 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.386952 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.388412 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.388878 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.389055 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.389569 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:46:26.390135 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:46:26.390307 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.390358 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.390466 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 18:46:26.394730 27571 tablet.cc:2369] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:46:26.399987 27525 raft_consensus.cc:1261] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 1 FOLLOWER]: Refusing update from remote peer 0cd88091f8c5425980986ee1f14e1d5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:26.399991 27248 raft_consensus.cc:1261] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 1 FOLLOWER]: Refusing update from remote peer 0cd88091f8c5425980986ee1f14e1d5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:46:26.401052 27600 consensus_queue.cc:1035] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b029536301247338d41b9c4278d9656" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 34631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:26.401443 27596 consensus_queue.cc:1035] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9bb072bd2c5415cb47b20ec2e713621" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 38087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:46:26.417402 27604 mvcc.cc:205] Tried to move back new op lower bound from 7298994939483467776 to 7298994939287326720. Current Snapshot: MvccSnapshot[applied={T|T < 7298994939483467776}]
I20260620 18:46:26.431109 27079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:26.431519 27079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:46:26.435953 27079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8be71ce6fbd64ae298b06e149df04702: Sending DeleteTablet for 3 replicas of tablet e044e0383a344f858467408ae82549cd
I20260620 18:46:26.437376 27228 tablet_service.cc:1514] Processing DeleteTablet for tablet e044e0383a344f858467408ae82549cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:26 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:45308
I20260620 18:46:26.437378 27371 tablet_service.cc:1514] Processing DeleteTablet for tablet e044e0383a344f858467408ae82549cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:26 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:53552
I20260620 18:46:26.437808 27505 tablet_service.cc:1514] Processing DeleteTablet for tablet e044e0383a344f858467408ae82549cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:46:26 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:46148
I20260620 18:46:26.440306 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 27160
I20260620 18:46:26.450977 27612 tablet_replica.cc:330] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: stopping tablet replica
I20260620 18:46:26.451602 27612 raft_consensus.cc:2229] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:26.452265 27612 raft_consensus.cc:2258] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:26.457414 27612 ts_tablet_manager.cc:1905] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:26.459374 27613 tablet_replica.cc:330] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: stopping tablet replica
I20260620 18:46:26.459899 27613 raft_consensus.cc:2229] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:46:26.460670 27613 raft_consensus.cc:2258] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:26.466270 27613 ts_tablet_manager.cc:1905] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:26.473785 27614 tablet_replica.cc:330] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: stopping tablet replica
I20260620 18:46:26.474291 27614 raft_consensus.cc:2229] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:46:26.474872 27614 raft_consensus.cc:2258] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:46:26.474869 27613 ts_tablet_manager.cc:1918] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:26.475050 27613 log.cc:1196] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/e044e0383a344f858467408ae82549cd
I20260620 18:46:26.475747 27613 ts_tablet_manager.cc:1939] T e044e0383a344f858467408ae82549cd P 0cd88091f8c5425980986ee1f14e1d5f: Deleting consensus metadata
I20260620 18:46:26.477821 27066 catalog_manager.cc:4841] TS 0cd88091f8c5425980986ee1f14e1d5f (127.15.102.194:36245): tablet e044e0383a344f858467408ae82549cd (table test-table [id=b02b5726eefa4c8f9a589bd7efec516e]) successfully deleted
I20260620 18:46:26.479199 27614 ts_tablet_manager.cc:1905] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:46:26.479950 27612 ts_tablet_manager.cc:1918] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:26.480213 27612 log.cc:1196] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/e044e0383a344f858467408ae82549cd
I20260620 18:46:26.480772 27612 ts_tablet_manager.cc:1939] T e044e0383a344f858467408ae82549cd P d9bb072bd2c5415cb47b20ec2e713621: Deleting consensus metadata
Jun 20 18:46:26 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:46:26.482882 27064 catalog_manager.cc:4841] TS d9bb072bd2c5415cb47b20ec2e713621 (127.15.102.193:38087): tablet e044e0383a344f858467408ae82549cd (table test-table [id=b02b5726eefa4c8f9a589bd7efec516e]) successfully deleted
I20260620 18:46:26.486069 27614 ts_tablet_manager.cc:1918] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:46:26.486248 27614 log.cc:1196] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/e044e0383a344f858467408ae82549cd
I20260620 18:46:26.486737 27614 ts_tablet_manager.cc:1939] T e044e0383a344f858467408ae82549cd P 6b029536301247338d41b9c4278d9656: Deleting consensus metadata
I20260620 18:46:26.488384 27064 catalog_manager.cc:4841] TS 6b029536301247338d41b9c4278d9656 (127.15.102.195:34631): tablet e044e0383a344f858467408ae82549cd (table test-table [id=b02b5726eefa4c8f9a589bd7efec516e]) successfully deleted
I20260620 18:46:26.491868 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.441596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.15.102.193:38087)
0620 18:46:26.441824 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.441830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.467560 (+ 25730us) server_negotiation.cc:407] Connection header received
0620 18:46:26.468108 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.468115 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.468209 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.468293 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:26.470126 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.471163 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.475178 (+ 4015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.475968 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.476075 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.484143 (+ 8068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:26.484159 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:26.484177 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:26.484233 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:26.488690 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.489527 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.489539 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.489543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.489715 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.490115 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.490121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.490125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.490606 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:26.490927 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.491154 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.491453 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:46:26.492545 27288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:26.606784 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 27160
I20260620 18:46:26.626233 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 27299
Jun 20 18:46:26 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:46:26.643290 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.627285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.15.102.194:36245)
0620 18:46:26.627475 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.627483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.627504 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:46:26.627754 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.627758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.627823 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.627940 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:26.629260 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.630171 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.631625 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.632059 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.632215 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.636806 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:26.636823 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:26.636852 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:26.636899 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:26.640186 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.641002 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.641011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.641013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.641164 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.641589 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.641593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.641595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.641997 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:26.642367 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.642597 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.642838 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:46:26.644071 27431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:26.758322 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 27299
I20260620 18:46:26.785959 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 27442
Jun 20 18:46:26 dist-test-slave-zfp8 krb5kdc[26604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:46:26.803835 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:46:26.786815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.15.102.195:34631)
0620 18:46:26.787037 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:46:26.787043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:46:26.787062 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:46:26.787210 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:46:26.787215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:46:26.787309 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:46:26.787415 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:46:26.788724 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.789476 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.790790 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:46:26.791198 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:46:26.791384 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:46:26.796849 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:46:26.796866 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:46:26.796883 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:46:26.796923 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:46:26.800241 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.801464 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.801472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.801475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.801622 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:46:26.802095 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:46:26.802099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:46:26.802101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:46:26.802540 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:46:26.802766 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 18:46:26.803219 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 18:46:26.803470 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 18:46:26.804548 27565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:26.902104 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 27442
I20260620 18:46:26.919059 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 27048
I20260620 18:46:26.920328 27110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:46:27.034494 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 27048
I20260620 18:46:27.057053 15771 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:46:27.402782 15771 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:46:27.403085 15771 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:46:27.749804 15771 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:46:27.750 UTC [26634] LOG: received smart shutdown request
2026-06-20 18:46:27.753 UTC [26634] LOG: background worker "logical replication launcher" (PID 26643) exited with exit code 1
2026-06-20 18:46:27.754 UTC [26638] LOG: shutting down
2026-06-20 18:46:27.911 UTC [26634] LOG: database system is shut down
2026-06-20T18:46:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49973 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:46:28 dist-test-slave-zfp8 krb5kdc[27628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:46:28 dist-test-slave-zfp8 krb5kdc[27628](info): set up 2 sockets
Jun 20 18:46:28 dist-test-slave-zfp8 krb5kdc[27628](info): commencing operation
krb5kdc: starting...
W20260620 18:46:29.143848 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:46:29 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:46:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.102.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.102.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/rangeradmin_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/rangeradmin_127.15.102.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.102.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.102.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/rangerlookup_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/rangerlookup_127.15.102.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.102.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.102.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/HTTP_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/HTTP_127.15.102.212@KRBTEST.COM.keytab.
I20260620 18:46:29.495462 15771 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskc6hSqA/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/postgres -l logfile start
2026-06-20 18:46:32.044 UTC [27658] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 18:46:32.044 UTC [27658] LOG: listening on IPv4 address "127.15.102.212", port 55359
2026-06-20 18:46:32.047 UTC [27658] LOG: listening on Unix socket "/tmp/.s.PGSQL.55359"
2026-06-20 18:46:32.069 UTC [27661] LOG: database system was shut down at 2026-06-20 18:46:30 UTC
2026-06-20 18:46:32.073 UTC [27658] LOG: database system is ready to accept connections
I20260620 18:46:33.047434 15771 mini_postgres.cc:96] Postgres bound to 55359
2026-06-20 18:46:33.065 UTC [27669] FATAL: database "slave" does not exist
127.15.102.212:55359 - accepting connections
I20260620 18:46:33.065960 15771 mini_ranger.cc:161] Starting Ranger...
I20260620 18:46:33.094157 15771 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 18:46:33.588894 15771 mini_ranger.cc:87] Created ranger Postgres database
I20260620 18:46:33.589048 15771 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-admin
2026-06-20 18:46:33,874 [I] DB FLAVOR :POSTGRES
2026-06-20 18:46:33,875 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:46:33,875 [I] Checking connection..
2026-06-20 18:46:33,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 18:46:34,141 [I] Checking connection passed.
2026-06-20 18:46:34,141 [I] --------- Verifying version history table ---------
2026-06-20 18:46:34,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:46:34,406 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:46:34,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:46:34,669 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 18:46:34,669 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 18:46:34,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 18:46:34.929 UTC [27733] WARNING: there is no transaction in progress
2026-06-20 18:46:34,936 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 18:46:34,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 18:46:35,188 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 18:46:35,189 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:46:35,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 18:46:35,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 18:46:35,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zfp8.c.gcp-upstream.internal','N') ;"
2026-06-20 18:46:35,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:46:36,250 [I] Table x_portal_user does not exist in database ranger
2026-06-20 18:46:36,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:46:36,530 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 18:46:36,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:46:36,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:46:37,034 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 18:46:37,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 18:46:38.104 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.121 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.138 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.154 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.171 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.189 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.248 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.256 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.271 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.284 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.295 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.309 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.319 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.326 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.333 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:38.340 UTC [27841] WARNING: there is no transaction in progress
2026-06-20 18:46:39,012 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 18:46:39,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 18:46:39,261 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 18:46:39,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 18:46:39,533 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 18:46:39,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 18:46:39,801 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 18:46:39,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 18:46:40,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 18:46:40,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zfp8.c.gcp-upstream.internal';"
2026-06-20 18:46:40,590 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 18:46:40,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 18:46:40,842 [I] DB_PATCHES have already been applied
I20260620 18:46:40.848366 15771 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-admin:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskc6hSqA/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 18:46:40.848460 15771 mini_ranger.cc:193] Using host: 127.15.102.212
I20260620 18:46:40.863695 15771 mini_ranger.cc:227] Ranger admin URL: http://127.15.102.212:41513
Jun 20, 2026 6:46:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:46:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:46:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:46:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:46:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:46:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:46:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 18:46:42.033911 15771 mini_ranger.cc:160] Time spent starting Ranger: real 8.968s user 0.000s sys 0.042s
Jun 20, 2026 6:47:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:47:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:47:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:47:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:47:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 18:47:03.459010 15771 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.102.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.102.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/rangerkms_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.102.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/rangerkms_127.15.102.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.102.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.102.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.102.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/HTTP_127.15.102.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.102.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/HTTP_127.15.102.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 18:47:03 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981223, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 18:47:03.635867 15771 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 18:47:03.655951 15771 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 18:47:04.462900 15771 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 18:47:04.463148 15771 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms
I20260620 18:47:04.463204 15771 mini_ranger_kms.cc:227] Using postgres at 127.15.102.212:55359
2026-06-20 18:47:04,821 [I] DB FLAVOR :POSTGRES
2026-06-20 18:47:04,821 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 18:47:04,822 [I] Checking connection
2026-06-20 18:47:04,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 18:47:05,087 [I] connection success
2026-06-20 18:47:05,088 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 18:47:05,088 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 18:47:05,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 18:47:05,361 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 18:47:05,361 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 18:47:05,362 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 18:47:05,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.102.212:55359/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 18:47:05,650 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 18:47:06.122710 15771 mini_ranger_kms.cc:310] Created kms service
I20260620 18:47:06.356532 15771 mini_ranger_kms.cc:326] Created kudu user
I20260620 18:47:06.403525 15771 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 18:47:06 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981223, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:06.803576 15771 mini_ranger_kms.cc:384] Added ranger policy
I20260620 18:47:06.803905 15771 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms:/tmp/dist-test-taskc6hSqA/build/asan/bin/postgresql.jar:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskc6hSqA/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 18:47:06.803972 15771 mini_ranger_kms.cc:240] Using host: 127.15.102.212
I20260620 18:47:06.821640 15771 mini_ranger_kms.cc:276] Ranger KMS PID: 28021
I20260620 18:47:06.821790 15771 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.102.212:36337
Jun 20, 2026 6:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 18:47:09.166246 15771 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.530s user 0.082s sys 0.288s
I20260620 18:47:09.166517 15771 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 18:47:09.166689 15771 mini_ranger_kms.cc:401] 127.15.102.212:36337/kms/v1/keys
Jun 20 18:47:09 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981229, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.102.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 18:47:09 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981229, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.102.212@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
Jun 20 18:47:10 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981229, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.102.212@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
Jun 20 18:47:12 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:47:12.545094 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46575
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:46575
--ranger_config_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:36337/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:12.891968 28072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:12.892323 28072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:12.892498 28072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:12.899900 28072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:12.900051 28072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:12.900169 28072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:12.900265 28072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:12.900355 28072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:12.910641 28072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.102.212:36337/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:46575
--ranger_config_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:12.912341 28072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:12.913858 28072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:12.926126 28077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:12.926204 28078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:12.926133 28080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:12.926571 28072 server_base.cc:1034] running on GCE node
I20260620 18:47:12.927738 28072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:12.929217 28072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:12.930454 28072 hybrid_clock.cc:648] HybridClock initialized: now 1781981232930397 us; error 54 us; skew 500 ppm
I20260620 18:47:12.930819 28072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:12 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981232, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:12.934069 28072 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:47:12.935770 28072 webserver.cc:460] Webserver started at http://127.15.102.254:34587/ using document root <none> and password file <none>
I20260620 18:47:12.936493 28072 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:12.936630 28072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:12.936959 28072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:47:12 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981232, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:12.997879 28072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "078b357d322a422182fab0e1a271f613"
format_stamp: "Formatted at 2026-06-20 18:47:12 on dist-test-slave-zfp8"
server_key: "46ca7120f506064c7ccde7cb84df9b6c"
server_key_iv: "dbebf24fe4f13c8c3cfce9506a648ebe"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:12.999023 28072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "078b357d322a422182fab0e1a271f613"
format_stamp: "Formatted at 2026-06-20 18:47:12 on dist-test-slave-zfp8"
server_key: "46ca7120f506064c7ccde7cb84df9b6c"
server_key_iv: "dbebf24fe4f13c8c3cfce9506a648ebe"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:13.006560 28072 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260620 18:47:13.024243 28087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:13.025882 28072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:47:13.026086 28072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "078b357d322a422182fab0e1a271f613"
format_stamp: "Formatted at 2026-06-20 18:47:12 on dist-test-slave-zfp8"
server_key: "46ca7120f506064c7ccde7cb84df9b6c"
server_key_iv: "dbebf24fe4f13c8c3cfce9506a648ebe"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:13.026337 28072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:13.050968 28072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:13.055655 28072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:13.056027 28072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:13.076664 28072 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:46575
I20260620 18:47:13.076668 28138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:46575 every 8 connection(s)
I20260620 18:47:13.078083 28072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:13.082054 28139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:13.083000 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 28072
I20260620 18:47:13.083288 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:47:13.084165 15771 external_mini_cluster.cc:1428] Setting key 6ce05b0adf2c2c6656e7cde1aef5b146
I20260620 18:47:13.100389 28139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Bootstrap starting.
I20260620 18:47:13.103965 28139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:13.105463 28139 log.cc:824] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:13.108906 28139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: No bootstrap required, opened a new log
Jun 20 18:47:13 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:13.118132 28139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } }
I20260620 18:47:13.118563 28139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:13.118672 28139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078b357d322a422182fab0e1a271f613, State: Initialized, Role: FOLLOWER
I20260620 18:47:13.119428 28139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } }
I20260620 18:47:13.119771 28139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:13.119959 28139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:13.120146 28139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:13.122362 28139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } }
I20260620 18:47:13.122910 28139 leader_election.cc:302] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 078b357d322a422182fab0e1a271f613; no voters:
I20260620 18:47:13.123797 28139 leader_election.cc:288] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:13.123906 28144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:13.124945 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:13.086364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43490 (local address 127.15.102.254:46575)
0620 18:47:13.099874 (+ 13510us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:13.099896 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:13.100662 (+ 766us) server_negotiation.cc:407] Connection header received
0620 18:47:13.103401 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:13.103434 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:13.104000 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:13.104498 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:13.106301 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:13.108551 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:13.110445 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:13.111604 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:13.111822 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:13.116607 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:13.116615 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:13.116633 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:13.116675 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:13.120348 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:13.121069 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:13.121079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:13.121088 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:13.121262 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:13.121678 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:13.121683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:13.121687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:13.122228 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:13.122559 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:13.122993 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 18:47:13.123453 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13243,"thread_start_us":12979,"threads_started":1}
I20260620 18:47:13.126451 28139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:13.127544 28144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 1 LEADER]: Becoming Leader. State: Replica: 078b357d322a422182fab0e1a271f613, State: Running, Role: LEADER
I20260620 18:47:13.128691 28144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } }
I20260620 18:47:13.132076 28145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "078b357d322a422182fab0e1a271f613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } } }
I20260620 18:47:13.132555 28145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:13.133618 28146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 078b357d322a422182fab0e1a271f613. Latest consensus state: current_term: 1 leader_uuid: "078b357d322a422182fab0e1a271f613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } } }
I20260620 18:47:13.134009 28146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:13.135010 28153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:13.139487 28153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:13.141695 28139 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 18:47:13.157186 28153 catalog_manager.cc:1347] Generated new cluster ID: 2c1f720328ad4ec1a92d924d1bda675d
I20260620 18:47:13.157349 28153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:13.172997 28153 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:13.175117 28153 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:13.184378 28153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Generated new TSK 0
I20260620 18:47:13.185499 28153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:47:13.914772 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:36337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46575
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:47:14 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981234, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 18:47:14.349996 28184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:14.350449 28184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:14.350610 28184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:14.362605 28184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:14.362774 28184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:14.362984 28184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:47:14.379281 28184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.102.212:36337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:14.381357 28184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:14.383260 28184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:14.401157 28194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:14.406456 28195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:14.411286 28184 server_base.cc:1034] running on GCE node
W20260620 18:47:14.411791 28197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:14.412736 28184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:14.416625 28184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:14.417959 28184 hybrid_clock.cc:648] HybridClock initialized: now 1781981234417896 us; error 55 us; skew 500 ppm
I20260620 18:47:14.418396 28184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:14 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981234, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:14.423030 28184 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:47:14.428750 28184 webserver.cc:460] Webserver started at http://127.15.102.193:38089/ using document root <none> and password file <none>
I20260620 18:47:14.429872 28184 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:14.430034 28184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:14.430449 28184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:47:14 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981234, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:14.450727 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "1cfcb6e137b34c81b31e41559056872a"
format_stamp: "Formatted at 2026-06-20 18:47:14 on dist-test-slave-zfp8"
server_key: "d2849f7fcb951b8be973268545363845"
server_key_iv: "db24fcfd5aa4ab27369a882c2046094b"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:14.452020 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "1cfcb6e137b34c81b31e41559056872a"
format_stamp: "Formatted at 2026-06-20 18:47:14 on dist-test-slave-zfp8"
server_key: "d2849f7fcb951b8be973268545363845"
server_key_iv: "db24fcfd5aa4ab27369a882c2046094b"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:14.463742 28184 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20260620 18:47:14.496271 28204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:14.499078 28184 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.007s sys 0.009s
I20260620 18:47:14.499346 28184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "1cfcb6e137b34c81b31e41559056872a"
format_stamp: "Formatted at 2026-06-20 18:47:14 on dist-test-slave-zfp8"
server_key: "d2849f7fcb951b8be973268545363845"
server_key_iv: "db24fcfd5aa4ab27369a882c2046094b"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:14.499744 28184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:14.526890 28184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:14.532248 28184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:14.532580 28184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:14.533636 28184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:14.535197 28184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:14.535315 28184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:47:14.535513 28184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:14.535584 28184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:14.611123 28184 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:36595
I20260620 18:47:14.611123 28316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:36595 every 8 connection(s)
I20260620 18:47:14.612996 28184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:47:14.614570 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 28184
I20260620 18:47:14.614794 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:47:14.615638 15771 external_mini_cluster.cc:1428] Setting key f8aeb555e1bf31a1c3590caf6f1c126f
Jun 20 18:47:14 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981234, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:14.653811 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:14.623466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:55115 (local address 127.15.102.254:46575)
0620 18:47:14.628066 (+ 4600us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:14.628072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:14.628094 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:47:14.628159 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:14.628165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:14.628267 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:14.628390 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:14.635547 (+ 7157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:14.636880 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:14.639553 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:14.640217 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:14.640332 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:14.645267 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:14.645275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:14.645281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:14.645337 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:14.649338 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:14.650444 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:14.650453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:14.650460 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:14.650625 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:14.651259 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:14.651268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:14.651274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:14.651687 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:14.651848 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:14.653054 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0620 18:47:14.653340 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4470,"thread_start_us":4348,"threads_started":1}
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:47:14.656469 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:14.622765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46575 (local address 127.15.102.193:55115)
0620 18:47:14.624270 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:14.624330 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:14.626839 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:14.631954 (+ 5115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:14.631971 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:14.632740 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:14.633688 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:14.633719 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:14.637128 (+ 3409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:14.637134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:14.639361 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:14.639378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:14.640740 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:14.640749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:14.641071 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:14.641948 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:14.641981 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:14.644941 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:14.649546 (+ 4605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:14.649560 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:14.649583 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:14.650290 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:14.650800 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:14.650808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:14.650814 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:14.651094 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:14.651948 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:14.651959 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:14.652515 (+ 556us) client_negotiation.cc:769] Sending connection context
0620 18:47:14.654149 (+ 1634us) client_negotiation.cc:240] Negotiation successful
0620 18:47:14.654793 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":1179,"threads_started":1}
I20260620 18:47:14.658543 28317 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
I20260620 18:47:14.658912 28317 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:14.659907 28317 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
I20260620 18:47:14.662658 28104 ts_manager.cc:194] Registered new tserver with Master: 1cfcb6e137b34c81b31e41559056872a (127.15.102.193:36595)
I20260620 18:47:14.664800 28104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:55115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:47:14.691861 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:36337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46575
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:47:14 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981234, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
Jun 20 18:47:15 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981234, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
W20260620 18:47:15.175539 28323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:15.176059 28323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:15.176249 28323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:15.187963 28323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:15.188207 28323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:15.188441 28323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:47:15.204250 28323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.102.212:36337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:15.206302 28323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:15.208125 28323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:15.237135 28330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:15.239200 28331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:15.248661 28333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:15.249586 28323 server_base.cc:1034] running on GCE node
I20260620 18:47:15.250630 28323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:15.272517 28323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:15.282038 28323 hybrid_clock.cc:648] HybridClock initialized: now 1781981235281914 us; error 109 us; skew 500 ppm
I20260620 18:47:15.282680 28323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:15 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:15.287297 28323 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:47:15.323748 28323 webserver.cc:460] Webserver started at http://127.15.102.194:43561/ using document root <none> and password file <none>
I20260620 18:47:15.324915 28323 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:15.325122 28323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:15.325502 28323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:47:15 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:15.350286 28323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "a1490f87e0174256b46539c94eb24c2b"
format_stamp: "Formatted at 2026-06-20 18:47:15 on dist-test-slave-zfp8"
server_key: "dfc4671844c0898fb3a7872fe5dd9ab4"
server_key_iv: "de96f450b9a24cd013e327f28de80cc3"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:15.351548 28323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "a1490f87e0174256b46539c94eb24c2b"
format_stamp: "Formatted at 2026-06-20 18:47:15 on dist-test-slave-zfp8"
server_key: "dfc4671844c0898fb3a7872fe5dd9ab4"
server_key_iv: "de96f450b9a24cd013e327f28de80cc3"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:15.370507 28323 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.009s sys 0.008s
I20260620 18:47:15.413535 28340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:15.415916 28323 fs_manager.cc:725] Time spent opening block manager: real 0.028s user 0.014s sys 0.012s
I20260620 18:47:15.416203 28323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "a1490f87e0174256b46539c94eb24c2b"
format_stamp: "Formatted at 2026-06-20 18:47:15 on dist-test-slave-zfp8"
server_key: "dfc4671844c0898fb3a7872fe5dd9ab4"
server_key_iv: "de96f450b9a24cd013e327f28de80cc3"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:15.416729 28323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:15.437978 28323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:15.442601 28323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:15.442920 28323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:15.444074 28323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:15.445660 28323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:15.445771 28323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:15.445902 28323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:15.445976 28323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:15.506364 28323 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:35207
I20260620 18:47:15.506385 28459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:35207 every 8 connection(s)
I20260620 18:47:15.507963 28323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:47:15.508760 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 28323
I20260620 18:47:15.508936 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:47:15.509434 15771 external_mini_cluster.cc:1428] Setting key f5ee4d326eeaa3a5998dad05cff7b09e
Jun 20 18:47:15 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:15.533422 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:15.511653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:56541 (local address 127.15.102.254:46575)
0620 18:47:15.512597 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:15.512602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:15.513208 (+ 606us) server_negotiation.cc:407] Connection header received
0620 18:47:15.515589 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:15.515595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:15.515696 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:15.515807 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:15.517566 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:15.518454 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:15.520257 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:15.520716 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:15.520866 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:15.525428 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:15.525435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:15.525440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:15.525480 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:15.529202 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:15.530334 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:15.530340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:15.530350 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:15.530506 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:15.531086 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:15.531092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:15.531095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:15.531519 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:15.531665 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:15.532795 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0620 18:47:15.533040 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":719,"threads_started":1}
I20260620 18:47:15.535197 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:15.512214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46575 (local address 127.15.102.194:56541)
0620 18:47:15.513041 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:15.513080 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:15.515038 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:15.516234 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:15.516252 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:15.516731 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:15.517374 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:15.517399 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:15.518626 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:15.518632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:15.520098 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:15.520112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:15.520965 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:15.520972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:15.521211 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:15.522004 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:15.522028 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:15.525134 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:15.529379 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:15.529392 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:15.529416 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:15.530165 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:15.530641 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:15.530649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:15.530655 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:15.530947 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:15.531664 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:15.531676 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:15.532282 (+ 606us) client_negotiation.cc:769] Sending connection context
0620 18:47:15.532649 (+ 367us) client_negotiation.cc:240] Negotiation successful
0620 18:47:15.533342 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":2953,"threads_started":1}
I20260620 18:47:15.537299 28460 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46575
I20260620 18:47:15.537701 28460 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:15.538815 28460 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
I20260620 18:47:15.540856 28104 ts_manager.cc:194] Registered new tserver with Master: a1490f87e0174256b46539c94eb24c2b (127.15.102.194:35207)
I20260620 18:47:15.541813 28104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:56541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:47:15.570739 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:36337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46575
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:15.669389 28317 heartbeater.cc:503] Master 127.15.102.254:46575 was elected leader, sending a full tablet report...
W20260620 18:47:15.905186 28466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:15.905540 28466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:15.905704 28466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:15.913312 28466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:15.913486 28466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:15.913671 28466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:47:15.924913 28466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.102.212:36337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:15.926406 28466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:15.928025 28466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:15.942198 28471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:15.942214 28472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:15.944416 28474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:15.945623 28466 server_base.cc:1034] running on GCE node
I20260620 18:47:15.946339 28466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:15.947642 28466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:15.948881 28466 hybrid_clock.cc:648] HybridClock initialized: now 1781981235948844 us; error 48 us; skew 500 ppm
I20260620 18:47:15.949251 28466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:15 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:15.952785 28466 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:47:15.954546 28466 webserver.cc:460] Webserver started at http://127.15.102.195:45831/ using document root <none> and password file <none>
I20260620 18:47:15.955407 28466 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:15.955560 28466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:15.955894 28466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 18:47:15 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:15.974103 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "45b90529fbb04dc6b6e13b058c7999a4"
format_stamp: "Formatted at 2026-06-20 18:47:15 on dist-test-slave-zfp8"
server_key: "1062c219971cd36d5f8ad2c9a846fc5c"
server_key_iv: "12a91090b681048bc117bf269618a3d1"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:15.975045 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "45b90529fbb04dc6b6e13b058c7999a4"
format_stamp: "Formatted at 2026-06-20 18:47:15 on dist-test-slave-zfp8"
server_key: "1062c219971cd36d5f8ad2c9a846fc5c"
server_key_iv: "12a91090b681048bc117bf269618a3d1"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:15.981220 28466 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 18:47:15.999603 28481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:16.001677 28466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260620 18:47:16.001907 28466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "45b90529fbb04dc6b6e13b058c7999a4"
format_stamp: "Formatted at 2026-06-20 18:47:15 on dist-test-slave-zfp8"
server_key: "1062c219971cd36d5f8ad2c9a846fc5c"
server_key_iv: "12a91090b681048bc117bf269618a3d1"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:16.002183 28466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:16.020996 28466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:16.026057 28466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:16.026502 28466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:16.027819 28466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:16.029630 28466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:16.029744 28466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:16.029865 28466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:16.029924 28466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:16.068454 28466 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:45333
I20260620 18:47:16.068488 28593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:45333 every 8 connection(s)
I20260620 18:47:16.069713 28466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:47:16.074420 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 28466
I20260620 18:47:16.074610 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:47:16.075062 15771 external_mini_cluster.cc:1428] Setting key 3a48e833bd36f94775a0f8e3826cd676
Jun 20 18:47:16 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:16.090953 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:16.072466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:34803 (local address 127.15.102.254:46575)
0620 18:47:16.073016 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:16.073021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:16.074030 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 18:47:16.076343 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:16.076351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:16.076464 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:16.076600 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:16.078555 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.079365 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.080942 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.081413 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.081542 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:16.084982 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:16.084987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:16.084991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:16.085026 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:16.087583 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.088297 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.088301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.088303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.088423 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.088823 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.088828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.088830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.089152 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:16.089329 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:16.090283 (+ 954us) server_negotiation.cc:299] Negotiation successful
0620 18:47:16.090557 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":303,"threads_started":1}
I20260620 18:47:16.091917 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:16.073145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46575 (local address 127.15.102.195:34803)
0620 18:47:16.073861 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:16.073901 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:16.075933 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:16.076868 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:16.076878 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:16.077495 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:16.078374 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:16.078404 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.079515 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.079521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:16.080784 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:16.080795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.081650 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.081657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:16.081898 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:16.082620 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:16.082643 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:16.084756 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:16.087721 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:16.087728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:16.087744 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:16.088190 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:16.088516 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:16.088519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:16.088521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:16.088714 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:16.089301 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:16.089307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:16.089774 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 18:47:16.090083 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 18:47:16.090573 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":292,"threads_started":1}
I20260620 18:47:16.093290 28594 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46575
I20260620 18:47:16.093642 28594 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:16.094430 28594 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
I20260620 18:47:16.095947 28104 ts_manager.cc:194] Registered new tserver with Master: 45b90529fbb04dc6b6e13b058c7999a4 (127.15.102.195:45333)
I20260620 18:47:16.096815 28104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:34803
I20260620 18:47:16.104498 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:47:16.106389 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 28184
Jun 20 18:47:16 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:47:16.124648 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:16.107549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.15.102.193:36595)
0620 18:47:16.108291 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:16.108299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:16.108326 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:47:16.108391 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:16.108398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:16.108652 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:16.108813 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:16.110359 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.111493 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.113102 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.113698 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.113837 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:16.118345 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:16.118362 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:16.118379 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:16.118425 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:16.121618 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.122439 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.122447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.122450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.122604 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.122988 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.122992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.122994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.123451 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:16.123686 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:16.124059 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 18:47:16.124293 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":356,"threads_started":1}
I20260620 18:47:16.126030 28312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:16.212077 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 28184
I20260620 18:47:16.228847 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 28323
Jun 20 18:47:16 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:47:16.246733 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:16.229895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.15.102.194:35207)
0620 18:47:16.230510 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:16.230516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:16.230543 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:47:16.230649 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:16.230659 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:16.230959 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:16.231155 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:16.232495 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.233531 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.234984 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.235681 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.235809 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:16.240552 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:16.240569 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:16.240587 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:16.240642 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:16.243779 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.244597 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.244606 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.244608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.244773 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.245193 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.245197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.245199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.245611 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:16.245871 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:16.246165 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 18:47:16.246413 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":321,"threads_started":1}
I20260620 18:47:16.248287 28455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:16.331991 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 28323
I20260620 18:47:16.348452 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 28466
Jun 20 18:47:16 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:47:16.370816 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:16.349521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.15.102.195:45333)
0620 18:47:16.350200 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:16.350207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:16.350237 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:47:16.350301 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:16.350308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:16.350539 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:16.350673 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:16.352174 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.353244 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.354681 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:16.355385 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:16.355489 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:16.364645 (+ 9156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:16.364662 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:16.364681 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:16.364730 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:16.367965 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.368818 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.368826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.368829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.368972 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:16.369325 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:16.369328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:16.369330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:16.369671 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:16.369931 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:16.370179 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 18:47:16.370381 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":366,"threads_started":1}
I20260620 18:47:16.372244 28589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:16.459585 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 28466
I20260620 18:47:16.476419 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 28072
I20260620 18:47:16.477824 28134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:16.587977 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 28072
I20260620 18:47:16.614125 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46575
--webserver_interface=127.15.102.254
--webserver_port=34587
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:46575
--ranger_config_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:36337/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.102.212:36337/kms with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:16.965731 28606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:16.966054 28606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:16.966213 28606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:16.973804 28606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:16.973951 28606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:16.974051 28606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:16.974161 28606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:16.974234 28606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:16.988565 28606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.102.212:36337/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:46575
--ranger_config_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:46575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=34587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:16.990074 28606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:16.991740 28606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:17.005167 28611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:17.005155 28614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:17.005240 28612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:17.006093 28606 server_base.cc:1034] running on GCE node
I20260620 18:47:17.007066 28606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:17.008945 28606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:17.010233 28606 hybrid_clock.cc:648] HybridClock initialized: now 1781981237010173 us; error 62 us; skew 500 ppm
I20260620 18:47:17.010649 28606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:17 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981237, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:17.014206 28606 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:47:17.016052 28606 webserver.cc:460] Webserver started at http://127.15.102.254:34587/ using document root <none> and password file <none>
I20260620 18:47:17.016856 28606 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:17.017025 28606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:47:17 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981237, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:17.106731 28606 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:47:17.110705 28622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:17.112596 28606 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 18:47:17.112819 28606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "078b357d322a422182fab0e1a271f613"
format_stamp: "Formatted at 2026-06-20 18:47:12 on dist-test-slave-zfp8"
server_key: "46ca7120f506064c7ccde7cb84df9b6c"
server_key_iv: "dbebf24fe4f13c8c3cfce9506a648ebe"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:17.113976 28606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:17.133394 28606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:17.138712 28606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:17.139020 28606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:17.160955 28606 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:46575
I20260620 18:47:17.161068 28673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:46575 every 8 connection(s)
I20260620 18:47:17.162799 28606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:17.170830 28674 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:47:17.171748 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 28606
I20260620 18:47:17.172838 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:36337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:36595
--local_ip_for_outbound_sockets=127.15.102.193
--tserver_master_addrs=127.15.102.254:46575
--webserver_port=38089
--webserver_interface=127.15.102.193
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.102.212:36337/kms with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:17.176723 28674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Bootstrap starting.
I20260620 18:47:17.199115 28674 log.cc:824] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:17.209476 28674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:47:17.210114 28674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Bootstrap complete.
I20260620 18:47:17.220186 28674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } }
I20260620 18:47:17.221223 28674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078b357d322a422182fab0e1a271f613, State: Initialized, Role: FOLLOWER
I20260620 18:47:17.222034 28674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } }
I20260620 18:47:17.222292 28674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:17.222412 28674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:17.222601 28674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:47:17.227381 28674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } }
I20260620 18:47:17.228137 28674 leader_election.cc:302] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 078b357d322a422182fab0e1a271f613; no voters:
I20260620 18:47:17.228893 28674 leader_election.cc:288] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:47:17.229049 28678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:47:17.230399 28678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [term 2 LEADER]: Becoming Leader. State: Replica: 078b357d322a422182fab0e1a271f613, State: Running, Role: LEADER
I20260620 18:47:17.231282 28678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } }
I20260620 18:47:17.231669 28674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:17.233989 28680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 078b357d322a422182fab0e1a271f613. Latest consensus state: current_term: 2 leader_uuid: "078b357d322a422182fab0e1a271f613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } } }
I20260620 18:47:17.234354 28680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:17.234925 28679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "078b357d322a422182fab0e1a271f613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078b357d322a422182fab0e1a271f613" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 46575 } } }
I20260620 18:47:17.235291 28679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:17.236219 28684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:17.254827 28684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:17.257159 28684 catalog_manager.cc:1259] Loaded cluster ID: 2c1f720328ad4ec1a92d924d1bda675d
I20260620 18:47:17.257437 28684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:17.263834 28684 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:17.266233 28684 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Loaded TSK: 0
I20260620 18:47:17.268959 28684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:17.275610 28674 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 18:47:17.676774 28676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:17.677347 28676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:17.677546 28676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:17.690088 28676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:17.690277 28676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:17.690546 28676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:47:17.707607 28676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.102.212:36337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:36595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=38089
--tserver_master_addrs=127.15.102.254:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:17.709771 28676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:17.711922 28676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:17.739362 28711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:17.740046 28712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:17.746088 28676 server_base.cc:1034] running on GCE node
W20260620 18:47:17.746632 28714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:17.747690 28676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:17.755571 28676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:17.759392 28676 hybrid_clock.cc:648] HybridClock initialized: now 1781981237759296 us; error 93 us; skew 500 ppm
I20260620 18:47:17.759938 28676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:17 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981237, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:17.764943 28676 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:47:17.778167 28676 webserver.cc:460] Webserver started at http://127.15.102.193:38089/ using document root <none> and password file <none>
I20260620 18:47:17.779314 28676 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:17.779603 28676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:47:17 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981237, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:17.812182 28676 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.002s sys 0.009s
I20260620 18:47:17.821480 28723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:17.824004 28676 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20260620 18:47:17.824291 28676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "1cfcb6e137b34c81b31e41559056872a"
format_stamp: "Formatted at 2026-06-20 18:47:14 on dist-test-slave-zfp8"
server_key: "d2849f7fcb951b8be973268545363845"
server_key_iv: "db24fcfd5aa4ab27369a882c2046094b"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:17.825874 28676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:17.887892 28676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:17.894855 28676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:17.895355 28676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:17.896770 28676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:17.898876 28676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:17.899003 28676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:17.899121 28676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:17.899219 28676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:17.964435 28676 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:36595
I20260620 18:47:17.964532 28835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:36595 every 8 connection(s)
I20260620 18:47:17.966117 28676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:47:17.973008 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 28676
I20260620 18:47:17.973718 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:36337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:35207
--local_ip_for_outbound_sockets=127.15.102.194
--tserver_master_addrs=127.15.102.254:46575
--webserver_port=43561
--webserver_interface=127.15.102.194
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.102.212:36337/kms with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:47:17 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981237, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:18.012956 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:17.969922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:48499 (local address 127.15.102.254:46575)
0620 18:47:17.970835 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:17.970858 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:17.971501 (+ 643us) server_negotiation.cc:407] Connection header received
0620 18:47:17.974783 (+ 3282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:17.974807 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:17.975486 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:17.975983 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:17.978122 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:17.980210 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:17.982501 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:17.983821 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:17.984546 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:17.989354 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:17.998287 (+ 8933us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:17.998329 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:17.998443 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:18.003370 (+ 4927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:18.004577 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:18.004588 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:18.004598 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:18.004828 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:18.005404 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:18.005410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:18.005414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:18.006114 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:18.006341 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:18.010851 (+ 4510us) server_negotiation.cc:299] Negotiation successful
0620 18:47:18.011424 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":291,"threads_started":1}
I20260620 18:47:18.014333 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:17.969955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46575 (local address 127.15.102.193:48499)
0620 18:47:17.971261 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:17.971332 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:17.973990 (+ 2658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:17.976211 (+ 2221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:17.976225 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:17.976939 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:17.977837 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:17.977865 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:17.980457 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:17.980465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:17.982285 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:17.982304 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:17.984291 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:17.984313 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:17.985068 (+ 755us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:17.986021 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:17.986057 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:17.988842 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:18.003636 (+ 14794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:18.003647 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:18.003667 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:18.004370 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:18.004972 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:18.004980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:18.004985 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:18.005267 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:18.006428 (+ 1161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:18.006437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:18.006880 (+ 443us) client_negotiation.cc:769] Sending connection context
0620 18:47:18.007191 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 18:47:18.007835 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":693,"threads_started":1}
I20260620 18:47:18.021669 28836 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46575
I20260620 18:47:18.022037 28836 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:18.022961 28836 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
W20260620 18:47:18.026309 28836 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:46575 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:47:18.026600 28836 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:18.026929 28836 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
I20260620 18:47:18.028427 28836 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:18.028750 28836 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
W20260620 18:47:18.029937 28836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:47:18.493116 28840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:18.493693 28840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:18.493912 28840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:18.507063 28840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:18.507341 28840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:18.507587 28840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:47:18.522492 28840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.102.212:36337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:35207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=43561
--tserver_master_addrs=127.15.102.254:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:18.524047 28840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:18.525547 28840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:18.569317 28857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:18.574061 28856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:18.577704 28840 server_base.cc:1034] running on GCE node
W20260620 18:47:18.577971 28860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:18.579025 28840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:18.583644 28840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:18.585053 28840 hybrid_clock.cc:648] HybridClock initialized: now 1781981238584967 us; error 71 us; skew 500 ppm
I20260620 18:47:18.585533 28840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:18 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:18.590560 28840 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:47:18.593070 28840 webserver.cc:460] Webserver started at http://127.15.102.194:43561/ using document root <none> and password file <none>
I20260620 18:47:18.594151 28840 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:18.594340 28840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:47:18 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
Jun 20 18:47:18 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781981238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:18.630190 28840 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.004s sys 0.008s
I20260620 18:47:18.634831 28867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:18.638280 28840 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260620 18:47:18.638626 28840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "a1490f87e0174256b46539c94eb24c2b"
format_stamp: "Formatted at 2026-06-20 18:47:15 on dist-test-slave-zfp8"
server_key: "dfc4671844c0898fb3a7872fe5dd9ab4"
server_key_iv: "de96f450b9a24cd013e327f28de80cc3"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:18.640492 28840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:18.674335 28840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:18.679378 28840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:18.679749 28840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:18.681046 28840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:18.682798 28840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:18.682897 28840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:18.683050 28840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:18.683146 28840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:18.737627 28840 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:35207
I20260620 18:47:18.737763 28980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:35207 every 8 connection(s)
I20260620 18:47:18.739642 28840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:47:18.742399 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 28840
I20260620 18:47:18.743604 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.102.212:36337/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:45333
--local_ip_for_outbound_sockets=127.15.102.195
--tserver_master_addrs=127.15.102.254:46575
--webserver_port=45831
--webserver_interface=127.15.102.195
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:39815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.102.212:36337/kms with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:47:18 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:18.795761 28982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:18.749936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46575 (local address 127.15.102.194:56241)
0620 18:47:18.769227 (+ 19291us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:18.769314 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:18.775714 (+ 6400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:18.777143 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:18.777154 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:18.777678 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:18.778389 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:18.778412 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:18.779788 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:18.779792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:18.781175 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:18.781190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:18.782087 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:18.782094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:18.782356 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:18.783134 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:18.783160 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:18.785583 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:18.790136 (+ 4553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:18.790151 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:18.790173 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:18.790877 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:18.791428 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:18.791433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:18.791437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:18.791735 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:18.792532 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:18.792541 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:18.793071 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 18:47:18.793472 (+ 401us) client_negotiation.cc:240] Negotiation successful
0620 18:47:18.794128 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18892,"thread_start_us":18604,"threads_started":1}
I20260620 18:47:18.797008 28983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:18.751742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:56241 (local address 127.15.102.254:46575)
0620 18:47:18.775752 (+ 24010us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:18.775760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:18.775797 (+ 37us) server_negotiation.cc:407] Connection header received
0620 18:47:18.776338 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:18.776345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:18.776447 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:18.776572 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:18.778627 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:18.779587 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:18.781380 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:18.781797 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:18.781946 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:18.785879 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:18.785885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:18.785889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:18.785929 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:18.789889 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:18.791068 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:18.791075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:18.791079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:18.791277 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:18.791897 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:18.791904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:18.791907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:18.792286 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:18.792430 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:18.796318 (+ 3888us) server_negotiation.cc:299] Negotiation successful
0620 18:47:18.796604 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23826,"thread_start_us":23681,"threads_started":1}
I20260620 18:47:18.799567 28981 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46575
I20260620 18:47:18.799950 28981 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:18.800958 28981 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
I20260620 18:47:18.803970 28639 ts_manager.cc:194] Registered new tserver with Master: a1490f87e0174256b46539c94eb24c2b (127.15.102.194:35207)
I20260620 18:47:18.806386 28639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:56241
I20260620 18:47:19.030118 28836 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:19.030604 28836 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
I20260620 18:47:19.032042 28639 ts_manager.cc:194] Registered new tserver with Master: 1cfcb6e137b34c81b31e41559056872a (127.15.102.193:36595)
I20260620 18:47:19.033334 28639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:48499
Jun 20 18:47:19 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
W20260620 18:47:19.277499 28984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:19.278041 28984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:19.278226 28984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:19.289327 28984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:19.289579 28984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:19.289788 28984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:47:19.307005 28984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:39815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.102.212:36337/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:45333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=45831
--tserver_master_addrs=127.15.102.254:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:19.309090 28984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:19.311131 28984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:19.352281 28991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:19.357110 28990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:19.359704 28984 server_base.cc:1034] running on GCE node
W20260620 18:47:19.360522 28994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:19.361617 28984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:19.369990 28984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:19.373569 28984 hybrid_clock.cc:648] HybridClock initialized: now 1781981239373458 us; error 105 us; skew 500 ppm
I20260620 18:47:19.374256 28984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:19 dist-test-slave-zfp8 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981239, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:19.379572 28984 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:47:19.400995 28984 webserver.cc:460] Webserver started at http://127.15.102.195:45831/ using document root <none> and password file <none>
I20260620 18:47:19.402390 28984 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:19.402707 28984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 18:47:19 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981239, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:19.477888 28984 fs_manager.cc:709] Time spent opening directory manager: real 0.017s user 0.005s sys 0.001s
I20260620 18:47:19.491221 29001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:19.493872 28984 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.001s
I20260620 18:47:19.494233 28984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "45b90529fbb04dc6b6e13b058c7999a4"
format_stamp: "Formatted at 2026-06-20 18:47:15 on dist-test-slave-zfp8"
server_key: "1062c219971cd36d5f8ad2c9a846fc5c"
server_key_iv: "12a91090b681048bc117bf269618a3d1"
server_key_version: "kuduclusterkey@0"
I20260620 18:47:19.496762 28984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:19.605775 28984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:19.617189 28984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:19.620783 28984 kserver.cc:163] Server-wide thread pool size limit: 3276
Jun 20 18:47:19 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for HTTP/127.15.102.212@KRBTEST.COM
I20260620 18:47:19.637665 28984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:19.648327 28984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:19.648553 28984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 18:47:19.648728 28984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:19.648840 28984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:19.812067 28981 heartbeater.cc:503] Master 127.15.102.254:46575 was elected leader, sending a full tablet report...
I20260620 18:47:20.037318 28836 heartbeater.cc:503] Master 127.15.102.254:46575 was elected leader, sending a full tablet report...
I20260620 18:47:20.154409 28984 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:45333
I20260620 18:47:20.154444 29121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:45333 every 8 connection(s)
I20260620 18:47:20.156661 28984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:47:20.161979 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 28984
Jun 20 18:47:20 dist-test-slave-zfp8 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981239, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:20.188344 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.159991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:34417 (local address 127.15.102.254:46575)
0620 18:47:20.160561 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.160569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.162683 (+ 2114us) server_negotiation.cc:407] Connection header received
0620 18:47:20.165094 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.165102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.165205 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.165315 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:20.168331 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.169978 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.173477 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.174463 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.174597 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.179594 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:20.179601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:20.179606 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:20.179657 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:20.183688 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.184788 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.184793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.184797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.184962 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.185823 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.185829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.185832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.186257 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:20.186424 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.187670 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.187986 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":5125,"threads_started":1}
I20260620 18:47:20.189478 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.160739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:46575 (local address 127.15.102.195:34417)
0620 18:47:20.162461 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.162528 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:20.164753 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:20.165895 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:20.165915 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:20.166816 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:20.168033 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.168067 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.170264 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.170272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.173139 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.173172 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.174954 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.174961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.175352 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.176392 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:20.176426 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:20.179323 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:20.183899 (+ 4576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:20.183910 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:20.183931 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:20.184600 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:20.185086 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:20.185090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:20.185094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:20.185673 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:20.186415 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:20.186424 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:20.187182 (+ 758us) client_negotiation.cc:769] Sending connection context
0620 18:47:20.187530 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 18:47:20.188089 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":379,"threads_started":1}
I20260620 18:47:20.191263 29122 heartbeater.cc:347] Connected to a master server at 127.15.102.254:46575
I20260620 18:47:20.191625 29122 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:20.192180 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.165321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.15.102.254:46575)
0620 18:47:20.175364 (+ 10043us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.175368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.175384 (+ 16us) server_negotiation.cc:407] Connection header received
0620 18:47:20.175435 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.175438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.175503 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.175639 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:20.177382 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.178546 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.180367 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.181030 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.181161 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.184792 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:20.184797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:20.184802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:20.184841 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:20.188006 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.189240 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.189246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.189249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.189395 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.189869 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.189873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.189875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.190234 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:20.190527 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.191620 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.191851 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9860,"thread_start_us":9698,"threads_started":1}
I20260620 18:47:20.192498 29122 heartbeater.cc:511] Master 127.15.102.254:46575 requested a full tablet report, sending...
I20260620 18:47:20.194314 28639 ts_manager.cc:194] Registered new tserver with Master: 45b90529fbb04dc6b6e13b058c7999a4 (127.15.102.195:45333)
I20260620 18:47:20.195564 28639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:34417
I20260620 18:47:20.196115 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:47:20.216471 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.201203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.15.102.254:46575)
0620 18:47:20.201465 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.201473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.201939 (+ 466us) server_negotiation.cc:407] Connection header received
0620 18:47:20.202094 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.202100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.202198 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.202318 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:20.203960 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.205024 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.206613 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.207286 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.207420 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.209963 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:20.209971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:20.209976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:20.210020 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:20.213457 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.214359 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.214366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.214370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.214545 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.214927 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.214933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.214936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.215323 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:20.215573 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.215909 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.216149 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:47:20.222190 28639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:47:20.230185 28639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:47:20.266021 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.253189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35207 (local address 127.0.0.1:50740)
0620 18:47:20.256147 (+ 2958us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.256167 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:20.257075 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:20.258109 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:20.258113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:20.258141 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:20.258491 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.258504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.260441 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.260446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.262813 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.262829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.264874 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.264884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.265108 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.265203 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 18:47:20.265330 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 18:47:20.265567 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2831,"mutex_wait_us":1,"spinlock_wait_cycles":1795328,"thread_start_us":2685,"threads_started":1}
I20260620 18:47:20.266178 29140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.248208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.15.102.195:45333)
0620 18:47:20.248999 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.249013 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.252039 (+ 3026us) server_negotiation.cc:407] Connection header received
0620 18:47:20.252759 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.252771 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.253175 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.253366 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:20.258025 (+ 4659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.259815 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.262912 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.264594 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.264776 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.264979 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.265631 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.265827 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":406,"threads_started":1}
I20260620 18:47:20.266997 29137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.247472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:45333 (local address 127.0.0.1:59718)
0620 18:47:20.251900 (+ 4428us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.251920 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:20.252220 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:20.253829 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:20.253841 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:20.257063 (+ 3222us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:20.257766 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.257781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.260014 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.260019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.262679 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.262702 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.265191 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.265198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.265386 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.265416 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:47:20.266390 (+ 974us) client_negotiation.cc:240] Negotiation successful
0620 18:47:20.266547 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4263,"mutex_wait_us":57,"spinlock_wait_cycles":71296,"thread_start_us":4101,"threads_started":1}
I20260620 18:47:20.268275 29141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.252432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.15.102.194:35207)
0620 18:47:20.256888 (+ 4456us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.256899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.256929 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:47:20.257325 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.257336 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.257625 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.257868 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:20.258718 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.260279 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.263064 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.264606 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.267475 (+ 2869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.267641 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.267726 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.267920 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4271,"thread_start_us":4071,"threads_started":1}
I20260620 18:47:20.268592 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.247044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:52108)
0620 18:47:20.249781 (+ 2737us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.252382 (+ 2601us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:20.252482 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:20.256975 (+ 4493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:20.256979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:20.257051 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:20.257731 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.257755 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.261351 (+ 3596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.261357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.263587 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.263602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.267728 (+ 4126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.267736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.267917 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.267950 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:47:20.268050 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 18:47:20.268194 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2372,"mutex_wait_us":41,"spinlock_wait_cycles":3328,"thread_start_us":2191,"threads_started":1}
I20260620 18:47:20.269243 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.247602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.15.102.193:36595)
0620 18:47:20.255922 (+ 8320us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.255933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.255965 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:47:20.256071 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.256084 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.256468 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.256713 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:20.258008 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.261175 (+ 3167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.265392 (+ 4217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.267094 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.267331 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.267553 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.268849 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.268996 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8094,"thread_start_us":7894,"threads_started":1}
I20260620 18:47:20.271682 29050 tablet_service.cc:1467] Processing CreateTablet for tablet ca8d94fe331f4a098350ea9b57ef41ec (DEFAULT_TABLE table=test-table [id=dcbb04bd1bf4458cb2c76921a8efd56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:20.272148 28916 tablet_service.cc:1467] Processing CreateTablet for tablet ca8d94fe331f4a098350ea9b57ef41ec (DEFAULT_TABLE table=test-table [id=dcbb04bd1bf4458cb2c76921a8efd56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:20.273181 29050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8d94fe331f4a098350ea9b57ef41ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:20.273674 28916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8d94fe331f4a098350ea9b57ef41ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:20.273357 28771 tablet_service.cc:1467] Processing CreateTablet for tablet ca8d94fe331f4a098350ea9b57ef41ec (DEFAULT_TABLE table=test-table [id=dcbb04bd1bf4458cb2c76921a8efd56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:20.274787 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8d94fe331f4a098350ea9b57ef41ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:20.297999 29144 tablet_bootstrap.cc:492] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Bootstrap starting.
I20260620 18:47:20.298772 29143 tablet_bootstrap.cc:492] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Bootstrap starting.
I20260620 18:47:20.303470 29144 tablet_bootstrap.cc:654] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:20.305029 29144 log.cc:824] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:20.308491 29143 tablet_bootstrap.cc:654] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:20.309516 29145 tablet_bootstrap.cc:492] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: Bootstrap starting.
I20260620 18:47:20.313683 29145 tablet_bootstrap.cc:654] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:20.313779 29144 tablet_bootstrap.cc:492] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: No bootstrap required, opened a new log
I20260620 18:47:20.314129 29144 ts_tablet_manager.cc:1397] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.010s
I20260620 18:47:20.315644 29145 log.cc:824] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:20.318622 29145 tablet_bootstrap.cc:492] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: No bootstrap required, opened a new log
I20260620 18:47:20.318931 29145 ts_tablet_manager.cc:1397] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 18:47:20.324182 29143 log.cc:824] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:20.324455 29144 raft_consensus.cc:348] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.325059 29144 raft_consensus.cc:374] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:20.325197 29144 raft_consensus.cc:729] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b90529fbb04dc6b6e13b058c7999a4, State: Initialized, Role: FOLLOWER
I20260620 18:47:20.326032 29144 consensus_queue.cc:260] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.328847 29145 raft_consensus.cc:348] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.329532 29145 raft_consensus.cc:374] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:20.329697 29145 raft_consensus.cc:729] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfcb6e137b34c81b31e41559056872a, State: Initialized, Role: FOLLOWER
I20260620 18:47:20.331610 29145 consensus_queue.cc:260] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.334617 29145 ts_tablet_manager.cc:1428] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260620 18:47:20.334739 29144 ts_tablet_manager.cc:1428] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Time spent starting tablet: real 0.020s user 0.012s sys 0.007s
I20260620 18:47:20.336249 29122 heartbeater.cc:503] Master 127.15.102.254:46575 was elected leader, sending a full tablet report...
I20260620 18:47:20.338443 29143 tablet_bootstrap.cc:492] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: No bootstrap required, opened a new log
I20260620 18:47:20.338862 29143 ts_tablet_manager.cc:1397] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Time spent bootstrapping tablet: real 0.042s user 0.013s sys 0.004s
I20260620 18:47:20.349547 29143 raft_consensus.cc:348] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.350140 29143 raft_consensus.cc:374] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:20.350244 29143 raft_consensus.cc:729] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1490f87e0174256b46539c94eb24c2b, State: Initialized, Role: FOLLOWER
I20260620 18:47:20.350963 29143 consensus_queue.cc:260] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.352903 29143 ts_tablet_manager.cc:1428] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260620 18:47:20.408944 29123 tablet.cc:2369] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:20.430051 29150 raft_consensus.cc:482] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:47:20.430421 29150 raft_consensus.cc:504] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.432659 29150 leader_election.cc:288] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45b90529fbb04dc6b6e13b058c7999a4 (127.15.102.195:45333), a1490f87e0174256b46539c94eb24c2b (127.15.102.194:35207)
I20260620 18:47:20.440845 29140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.433095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:44715 (local address 127.15.102.195:45333)
0620 18:47:20.433312 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.433318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.434164 (+ 846us) server_negotiation.cc:407] Connection header received
0620 18:47:20.434465 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.434471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.434563 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.434664 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:20.435584 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.436575 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.438748 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.439814 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.440001 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.440097 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.440411 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.440567 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:47:20.440904 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.433122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:45333 (local address 127.15.102.193:44715)
0620 18:47:20.434041 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.434064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:20.434230 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:20.434862 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:20.434867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:20.434899 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:20.435415 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.435427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.436727 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.436734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.438555 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.438576 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.440048 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.440054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.440222 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.440258 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:47:20.440355 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:47:20.440495 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":1,"spinlock_wait_cycles":28928,"thread_start_us":527,"threads_started":1}
I20260620 18:47:20.442063 29153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.434245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35207 (local address 127.15.102.193:34497)
0620 18:47:20.434899 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.434917 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:20.435130 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:20.435543 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:20.435548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:20.435584 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:20.435971 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.435980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.437328 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.437333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.439381 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:20.439396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.441236 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.441241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:20.441408 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.441441 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:47:20.441537 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:47:20.441670 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":4,"thread_start_us":557,"threads_started":1}
I20260620 18:47:20.442420 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8d94fe331f4a098350ea9b57ef41ec" candidate_uuid: "1cfcb6e137b34c81b31e41559056872a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b90529fbb04dc6b6e13b058c7999a4" is_pre_election: true
I20260620 18:47:20.442878 29141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.434573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:34497 (local address 127.15.102.194:35207)
0620 18:47:20.434809 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.434815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.435002 (+ 187us) server_negotiation.cc:407] Connection header received
0620 18:47:20.435337 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.435344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.435418 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.435524 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:20.436146 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.437154 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.439545 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.440842 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.441226 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.441330 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.441597 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.442602 (+ 1005us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:47:20.443089 29076 raft_consensus.cc:2454] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfcb6e137b34c81b31e41559056872a in term 0.
I20260620 18:47:20.443882 28726 leader_election.cc:302] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfcb6e137b34c81b31e41559056872a, 45b90529fbb04dc6b6e13b058c7999a4; no voters:
I20260620 18:47:20.444346 29150 raft_consensus.cc:2781] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:47:20.444536 29150 raft_consensus.cc:482] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:47:20.444371 28936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8d94fe331f4a098350ea9b57ef41ec" candidate_uuid: "1cfcb6e137b34c81b31e41559056872a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1490f87e0174256b46539c94eb24c2b" is_pre_election: true
I20260620 18:47:20.444674 29150 raft_consensus.cc:3037] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:20.445036 28936 raft_consensus.cc:2454] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfcb6e137b34c81b31e41559056872a in term 0.
I20260620 18:47:20.447680 29150 raft_consensus.cc:504] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.448412 29150 leader_election.cc:288] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [CANDIDATE]: Term 1 election: Requested vote from peers 45b90529fbb04dc6b6e13b058c7999a4 (127.15.102.195:45333), a1490f87e0174256b46539c94eb24c2b (127.15.102.194:35207)
I20260620 18:47:20.448876 29076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8d94fe331f4a098350ea9b57ef41ec" candidate_uuid: "1cfcb6e137b34c81b31e41559056872a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45b90529fbb04dc6b6e13b058c7999a4"
I20260620 18:47:20.449030 28936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8d94fe331f4a098350ea9b57ef41ec" candidate_uuid: "1cfcb6e137b34c81b31e41559056872a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1490f87e0174256b46539c94eb24c2b"
I20260620 18:47:20.449136 29076 raft_consensus.cc:3037] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:20.449259 28936 raft_consensus.cc:3037] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:20.453428 28936 raft_consensus.cc:2454] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfcb6e137b34c81b31e41559056872a in term 1.
I20260620 18:47:20.454938 28726 leader_election.cc:302] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfcb6e137b34c81b31e41559056872a, a1490f87e0174256b46539c94eb24c2b; no voters:
I20260620 18:47:20.455492 29150 raft_consensus.cc:2781] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:20.455847 29076 raft_consensus.cc:2454] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfcb6e137b34c81b31e41559056872a in term 1.
I20260620 18:47:20.456562 29150 raft_consensus.cc:686] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 1 LEADER]: Becoming Leader. State: Replica: 1cfcb6e137b34c81b31e41559056872a, State: Running, Role: LEADER
I20260620 18:47:20.457674 29150 consensus_queue.cc:237] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } }
I20260620 18:47:20.465188 28638 catalog_manager.cc:5495] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cfcb6e137b34c81b31e41559056872a (127.15.102.193). New cstate: current_term: 1 leader_uuid: "1cfcb6e137b34c81b31e41559056872a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfcb6e137b34c81b31e41559056872a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 36595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 } health_report { overall_health: UNKNOWN } } }
W20260620 18:47:20.471812 28837 tablet.cc:2369] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:20.504748 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.498459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.15.102.193:36595)
0620 18:47:20.498663 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.498668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.498687 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:47:20.498813 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.498817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.498898 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.499043 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:20.499548 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.500296 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.502319 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.502734 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.502841 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.503530 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:20.504097 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:20.504213 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.504368 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.504450 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:47:20.515120 28936 raft_consensus.cc:1261] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 1 FOLLOWER]: Refusing update from remote peer 1cfcb6e137b34c81b31e41559056872a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:20.515182 29076 raft_consensus.cc:1261] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 1 FOLLOWER]: Refusing update from remote peer 1cfcb6e137b34c81b31e41559056872a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:20.516348 29154 consensus_queue.cc:1035] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [LEADER]: Connected to new peer: Peer: permanent_uuid: "45b90529fbb04dc6b6e13b058c7999a4" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:20.516772 29150 consensus_queue.cc:1035] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1490f87e0174256b46539c94eb24c2b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:47:20.527407 28985 tablet.cc:2369] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:20.541183 29158 mvcc.cc:205] Tried to move back new op lower bound from 7298995161138020352 to 7298995160920096768. Current Snapshot: MvccSnapshot[applied={T|T < 7298995161138020352}]
I20260620 18:47:20.568305 28638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:47:20.568634 28638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:47:20.575510 28638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 078b357d322a422182fab0e1a271f613: Sending DeleteTablet for 3 replicas of tablet ca8d94fe331f4a098350ea9b57ef41ec
I20260620 18:47:20.576882 29050 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8d94fe331f4a098350ea9b57ef41ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:20 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:59718
I20260620 18:47:20.576905 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8d94fe331f4a098350ea9b57ef41ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:20 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:52108
I20260620 18:47:20.578939 28916 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8d94fe331f4a098350ea9b57ef41ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:20 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:50740
I20260620 18:47:20.592631 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 28676
I20260620 18:47:20.602705 29169 tablet_replica.cc:330] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: stopping tablet replica
I20260620 18:47:20.603094 29168 tablet_replica.cc:330] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: stopping tablet replica
I20260620 18:47:20.603765 29169 raft_consensus.cc:2229] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:47:20.604425 29168 raft_consensus.cc:2229] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:47:20.605294 29168 raft_consensus.cc:2258] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:20.606010 29169 raft_consensus.cc:2258] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:20.606660 29167 tablet_replica.cc:330] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: stopping tablet replica
I20260620 18:47:20.607187 29167 raft_consensus.cc:2229] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:47:20.608551 29167 raft_consensus.cc:2258] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:20.609957 29169 ts_tablet_manager.cc:1905] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:20.610849 29167 ts_tablet_manager.cc:1905] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:20.610994 29168 ts_tablet_manager.cc:1905] T ca8d94fe331f4a098350ea9b57ef41ec P 1cfcb6e137b34c81b31e41559056872a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:20.615908 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.594120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.15.102.193:36595)
0620 18:47:20.594353 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.594361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.594454 (+ 93us) server_negotiation.cc:407] Connection header received
0620 18:47:20.594648 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.594653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.594737 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.594855 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:20.596852 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.598105 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.600935 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.601461 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.601646 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.606218 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:20.606231 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:20.606246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:20.606295 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:20.610509 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.612223 (+ 1714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.612233 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.612237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.612483 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.613065 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.613070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.613074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.613524 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:20.613954 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.615019 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.615402 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:47:20.617893 29169 ts_tablet_manager.cc:1918] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:20.618045 28831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:20.618144 29169 log.cc:1196] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/ca8d94fe331f4a098350ea9b57ef41ec
I20260620 18:47:20.618796 29169 ts_tablet_manager.cc:1939] T ca8d94fe331f4a098350ea9b57ef41ec P a1490f87e0174256b46539c94eb24c2b: Deleting consensus metadata
I20260620 18:47:20.621394 28625 catalog_manager.cc:4841] TS a1490f87e0174256b46539c94eb24c2b (127.15.102.194:35207): tablet ca8d94fe331f4a098350ea9b57ef41ec (table test-table [id=dcbb04bd1bf4458cb2c76921a8efd56a]) successfully deleted
I20260620 18:47:20.624393 29167 ts_tablet_manager.cc:1918] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:20.624603 29167 log.cc:1196] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/ca8d94fe331f4a098350ea9b57ef41ec
I20260620 18:47:20.625078 29167 ts_tablet_manager.cc:1939] T ca8d94fe331f4a098350ea9b57ef41ec P 45b90529fbb04dc6b6e13b058c7999a4: Deleting consensus metadata
I20260620 18:47:20.626798 28625 catalog_manager.cc:4841] TS 45b90529fbb04dc6b6e13b058c7999a4 (127.15.102.195:45333): tablet ca8d94fe331f4a098350ea9b57ef41ec (table test-table [id=dcbb04bd1bf4458cb2c76921a8efd56a]) successfully deleted
I20260620 18:47:20.721763 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 28676
W20260620 18:47:20.740437 28626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:36595 (error 108)
I20260620 18:47:20.740746 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 28840
I20260620 18:47:20.742887 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.742321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:60070)
0620 18:47:20.742537 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.742689 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 18:47:20.743337 28626 catalog_manager.cc:4584] TS 1cfcb6e137b34c81b31e41559056872a (127.15.102.193:36595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca8d94fe331f4a098350ea9b57ef41ec: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
I20260620 18:47:20.757740 29141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.741855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.15.102.194:35207)
0620 18:47:20.742035 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.742041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.742079 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:47:20.742334 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.742341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.742451 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.742586 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:20.744444 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.745325 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.747442 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.747927 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.748041 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.751445 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:20.751461 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:20.751478 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:20.751517 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:20.754825 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.755691 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.755699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.755702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.755838 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.756194 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.756197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.756199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.756585 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:20.756846 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.757126 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.757308 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:47:20.758683 28976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:20.797212 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.796676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:60078)
0620 18:47:20.796939 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.797061 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 18:47:20.798425 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.798065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:60090)
0620 18:47:20.798210 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.798300 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:47:20.851455 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 28840
I20260620 18:47:20.859280 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.858735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:60096)
0620 18:47:20.858960 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.859054 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 18:47:20.860507 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.860129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:60102)
0620 18:47:20.860296 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.860376 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 18:47:20.869781 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 28984
I20260620 18:47:20.885506 29140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.870678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.15.102.195:45333)
0620 18:47:20.870917 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:20.870923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:20.870945 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:47:20.871131 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:20.871140 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:20.871217 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:20.871382 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:20.872951 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.874170 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.875770 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:20.876213 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:20.876369 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:20.879177 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:20.879195 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:20.879213 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:20.879314 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:20.882218 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.883069 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.883080 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.883084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.883326 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:20.883762 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:20.883768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:20.883771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:20.884194 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:20.884519 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:20.884874 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 18:47:20.885133 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:47:20.886268 29117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:20.941385 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.940815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:60110)
0620 18:47:20.941070 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.941191 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 18:47:20.942576 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:20.942127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:60126)
0620 18:47:20.942354 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:20.942435 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 18:47:20.979753 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 28984
I20260620 18:47:20.997408 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 28606
I20260620 18:47:20.998775 28669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:21.117736 29139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:21.117156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:36595 (local address 127.0.0.1:60128)
0620 18:47:21.117379 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:21.117510 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:36595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:47:21.117702 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 28606
I20260620 18:47:21.141711 15771 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 18:47:21.478229 15771 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 18:47:21.478498 15771 mini_ranger.cc:66] Stopping Ranger...
I20260620 18:47:21.825330 15771 mini_ranger.cc:68] Stopped Ranger
2026-06-20 18:47:21.825 UTC [27658] LOG: received smart shutdown request
2026-06-20 18:47:21.828 UTC [27658] LOG: background worker "logical replication launcher" (PID 27667) exited with exit code 1
2026-06-20 18:47:21.829 UTC [27662] LOG: shutting down
2026-06-20 18:47:22.004 UTC [27658] LOG: database system is shut down
2026-06-20T18:47:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54097 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:22 dist-test-slave-zfp8 krb5kdc[29183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:22 dist-test-slave-zfp8 krb5kdc[29183](info): set up 2 sockets
Jun 20 18:47:22 dist-test-slave-zfp8 krb5kdc[29183](info): commencing operation
krb5kdc: starting...
W20260620 18:47:23.236163 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:47:23 dist-test-slave-zfp8 krb5kdc[29183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:47:23.480162 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37963
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:37963,127.15.102.253:38359,127.15.102.252:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:23.828508 29196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:23.828918 29196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:23.829089 29196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:23.836822 29196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:23.836964 29196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:23.837055 29196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:23.837149 29196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:23.837244 29196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:23.847792 29196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:37963,127.15.102.253:38359,127.15.102.252:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:23.849334 29196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:23.850798 29196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:23.864087 29201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:23.864046 29202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:23.864027 29204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:23.865173 29196 server_base.cc:1034] running on GCE node
I20260620 18:47:23.866168 29196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:23.867938 29196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:23.869185 29196 hybrid_clock.cc:648] HybridClock initialized: now 1781981243869148 us; error 52 us; skew 500 ppm
I20260620 18:47:23.869583 29196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:23 dist-test-slave-zfp8 krb5kdc[29183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:23.873137 29196 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:47:23.874879 29196 webserver.cc:460] Webserver started at http://127.15.102.254:43071/ using document root <none> and password file <none>
I20260620 18:47:23.875738 29196 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:23.875893 29196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:23.876217 29196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:23.879500 29196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "bf454adc8bd74cd7bc6fba734628ed6e"
format_stamp: "Formatted at 2026-06-20 18:47:23 on dist-test-slave-zfp8"
server_key: "6d25b271f6bba6e49ecc5ed5a7ebf203"
server_key_iv: "8a16994c028541905a86b697a76a068b"
server_key_version: "encryptionkey@0"
I20260620 18:47:23.880398 29196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "bf454adc8bd74cd7bc6fba734628ed6e"
format_stamp: "Formatted at 2026-06-20 18:47:23 on dist-test-slave-zfp8"
server_key: "6d25b271f6bba6e49ecc5ed5a7ebf203"
server_key_iv: "8a16994c028541905a86b697a76a068b"
server_key_version: "encryptionkey@0"
I20260620 18:47:23.886317 29196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:47:23.890118 29211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:23.891641 29196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:47:23.891858 29196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "bf454adc8bd74cd7bc6fba734628ed6e"
format_stamp: "Formatted at 2026-06-20 18:47:23 on dist-test-slave-zfp8"
server_key: "6d25b271f6bba6e49ecc5ed5a7ebf203"
server_key_iv: "8a16994c028541905a86b697a76a068b"
server_key_version: "encryptionkey@0"
I20260620 18:47:23.892076 29196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:23.923640 29196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:23.928470 29196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:23.928794 29196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:23.948583 29196 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:37963
I20260620 18:47:23.948625 29262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:37963 every 8 connection(s)
I20260620 18:47:23.950088 29196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:23.954288 29263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:23.955649 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 29196
I20260620 18:47:23.955899 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:47:23.956753 15771 external_mini_cluster.cc:1428] Setting key 470f985bdc918cceb4e674ff8dc1d829
I20260620 18:47:23.965176 29263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.102.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.253@KRBTEST.COM" created.
Jun 20 18:47:23 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:23.993317 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:23.968981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37963 (local address 127.0.0.1:33800)
0620 18:47:23.970023 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:23.970080 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:23.973283 (+ 3203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:23.974443 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:23.974453 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:23.975507 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:23.976105 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:23.976139 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:23.977891 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:23.977899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:23.979962 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:23.979977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:23.981001 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:23.981008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:23.981177 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:23.981919 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:23.981957 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:23.985091 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:23.988763 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:23.988781 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:23.988798 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:23.989436 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:23.989903 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:23.989911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:23.989917 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:23.990150 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:23.991017 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:23.991028 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:23.991349 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 18:47:23.991676 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 18:47:23.992157 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":429,"threads_started":1}
I20260620 18:47:23.993343 29267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:23.968981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.15.102.254:37963)
0620 18:47:23.971158 (+ 2177us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:23.971167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:23.971212 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:47:23.973833 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:23.973840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:23.974152 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:23.974337 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:23.976376 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:23.977645 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:23.980122 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:23.980727 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:23.980955 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:23.985401 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:23.985411 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:23.985426 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:23.985462 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:23.988574 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:23.989600 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:23.989607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:23.989609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:23.989771 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:23.990293 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:23.990297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:23.990299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:23.990853 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:23.991038 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:23.991859 (+ 821us) server_negotiation.cc:299] Negotiation successful
0620 18:47:23.992379 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":1967,"spinlock_wait_cycles":61696,"thread_start_us":1705,"threads_started":1}
I20260620 18:47:23.996893 29263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:23.998071 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:23.997632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50616)
0620 18:47:23.997830 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:23.997940 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 18:47:23.998370 29212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
I20260620 18:47:24.000140 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:23.999751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50624)
0620 18:47:23.999916 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:23.999994 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:47:24.000366 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:38359: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab.
Entry for principal kudu/127.15.102.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab.
I20260620 18:47:24.014210 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.253:38359
--webserver_interface=127.15.102.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab
--principal=kudu/127.15.102.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:37963,127.15.102.253:38359,127.15.102.252:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:24.049717 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } attempt: 1
I20260620 18:47:24.051003 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.050513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50626)
0620 18:47:24.050733 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.050851 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:47:24.052206 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.051795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50640)
0620 18:47:24.051957 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.052047 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:47:24.052421 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:38359: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
I20260620 18:47:24.120795 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } attempt: 2
I20260620 18:47:24.122185 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.121630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50650)
0620 18:47:24.121915 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.122016 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 18:47:24.123464 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.122975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50664)
0620 18:47:24.123144 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.123300 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:47:24.123664 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:38359: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
I20260620 18:47:24.214944 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } attempt: 3
I20260620 18:47:24.216274 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.215801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50680)
0620 18:47:24.216026 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.216127 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:47:24.217403 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.217000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50692)
0620 18:47:24.217156 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.217256 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:47:24.217615 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:38359: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
I20260620 18:47:24.360877 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } attempt: 4
I20260620 18:47:24.362406 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.361677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50702)
0620 18:47:24.362040 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.362248 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260620 18:47:24.363611 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.363174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50704)
0620 18:47:24.363372 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.363479 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 18:47:24.363833 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:38359: Network error: Client connection negotiation failed: client connection to 127.15.102.253:38359: connect: Connection refused (error 111)
W20260620 18:47:24.366714 29270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:24.367080 29270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:24.367276 29270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:24.375592 29270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:24.375759 29270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:24.375862 29270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:24.375967 29270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:24.376055 29270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:24.387046 29270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:37963,127.15.102.253:38359,127.15.102.252:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab
--principal=kudu/127.15.102.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.253:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.102.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:24.388718 29270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:24.390156 29270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:24.403100 29276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:24.403158 29278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:24.403204 29275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:24.404270 29270 server_base.cc:1034] running on GCE node
I20260620 18:47:24.405167 29270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:24.406829 29270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:24.408123 29270 hybrid_clock.cc:648] HybridClock initialized: now 1781981244408061 us; error 57 us; skew 500 ppm
I20260620 18:47:24.408571 29270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:24 dist-test-slave-zfp8 krb5kdc[29183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:24.412317 29270 init.cc:377] Logged in from keytab as kudu/127.15.102.253@KRBTEST.COM (short username kudu)
I20260620 18:47:24.414093 29270 webserver.cc:460] Webserver started at http://127.15.102.253:32905/ using document root <none> and password file <none>
I20260620 18:47:24.414939 29270 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:24.415097 29270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:24.415524 29270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:24.419216 29270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/data/instance:
uuid: "4054b15c16274bcd9b31752e95da9cf6"
format_stamp: "Formatted at 2026-06-20 18:47:24 on dist-test-slave-zfp8"
server_key: "456285d2dd24455525a889656204ca17"
server_key_iv: "de76bf37ca8e1a704a63b6bb2cb556c4"
server_key_version: "encryptionkey@0"
I20260620 18:47:24.420337 29270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/wal/instance:
uuid: "4054b15c16274bcd9b31752e95da9cf6"
format_stamp: "Formatted at 2026-06-20 18:47:24 on dist-test-slave-zfp8"
server_key: "456285d2dd24455525a889656204ca17"
server_key_iv: "de76bf37ca8e1a704a63b6bb2cb556c4"
server_key_version: "encryptionkey@0"
I20260620 18:47:24.426620 29270 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 18:47:24.431073 29285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:24.432958 29270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:47:24.433235 29270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/wal
uuid: "4054b15c16274bcd9b31752e95da9cf6"
format_stamp: "Formatted at 2026-06-20 18:47:24 on dist-test-slave-zfp8"
server_key: "456285d2dd24455525a889656204ca17"
server_key_iv: "de76bf37ca8e1a704a63b6bb2cb556c4"
server_key_version: "encryptionkey@0"
I20260620 18:47:24.433490 29270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:24.461695 29270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:24.466774 29270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:24.467130 29270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:24.487561 29270 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.253:38359
I20260620 18:47:24.487663 29336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.253:38359 every 8 connection(s)
I20260620 18:47:24.489195 29270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
I20260620 18:47:24.493424 29337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:24.499048 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 29270
I20260620 18:47:24.499354 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-1/wal/instance
I20260620 18:47:24.499845 15771 external_mini_cluster.cc:1428] Setting key 6f48aff8f70e6f7f0f82a34f482ee03d
I20260620 18:47:24.501067 29337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:24 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.102.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.252@KRBTEST.COM" created.
I20260620 18:47:24.526752 29339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.504037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.15.102.254:37963)
0620 18:47:24.504650 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:24.504657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:24.506099 (+ 1442us) server_negotiation.cc:407] Connection header received
0620 18:47:24.508332 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:24.508341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:24.508453 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:24.508597 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:24.510374 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.511743 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.513503 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.514295 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.514485 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:24.518798 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:24.518808 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:24.518814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:24.518870 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:24.522629 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:24.523843 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:24.523849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:24.523851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:24.524009 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:24.524672 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:24.524680 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:24.524686 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:24.525034 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:24.525166 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:24.526254 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0620 18:47:24.526451 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":308,"threads_started":1}
I20260620 18:47:24.528540 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.504606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37963 (local address 127.0.0.1:33802)
0620 18:47:24.505576 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.505636 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:24.507958 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:24.508856 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:24.508866 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:24.509471 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:24.510134 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:24.510155 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.511945 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.511949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:24.513302 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:24.513317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.514598 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.514604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:24.514832 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:24.515724 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:24.515752 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:24.518461 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:24.522854 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:24.522865 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:24.522882 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:24.523633 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:24.524138 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:24.524142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:24.524170 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:24.524512 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:24.525209 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:24.525222 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:24.525785 (+ 563us) client_negotiation.cc:769] Sending connection context
0620 18:47:24.526145 (+ 360us) client_negotiation.cc:240] Negotiation successful
0620 18:47:24.526801 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":405,"threads_started":1}
I20260620 18:47:24.530678 29337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:24 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:47:24.548967 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.531557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50718)
0620 18:47:24.531788 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.531808 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:24.531932 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:24.533164 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:24.533171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:24.533751 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:24.533951 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:24.533984 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.536245 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.536250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:24.537126 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:24.537138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.538591 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.538600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:24.538837 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:24.539583 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:24.539602 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:24.542091 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:24.546022 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:24.546029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:24.546034 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:24.546644 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:24.547119 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:24.547126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:24.547132 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:24.547363 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:24.548037 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:24.548042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:24.548220 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 18:47:24.548354 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 18:47:24.548592 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 18:47:24.549165 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.531860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.15.102.253:38359)
0620 18:47:24.532669 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:24.532676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:24.532706 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:47:24.532757 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:24.532764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:24.532980 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:24.533106 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:24.534157 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.536062 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.537298 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.538378 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.538539 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:24.542368 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:24.542384 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:24.542400 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:24.542440 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:24.545816 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:24.546823 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:24.546832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:24.546834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:24.546967 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:24.547493 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:24.547497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:24.547499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:24.547892 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:24.548074 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:24.548509 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 18:47:24.548750 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":429,"threads_started":1}
I20260620 18:47:24.552086 29337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:24.553411 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.552818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52540)
0620 18:47:24.553030 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.553230 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:47:24.553828 29286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab.
Entry for principal kudu/127.15.102.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab.
I20260620 18:47:24.555722 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.555215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52548)
0620 18:47:24.555456 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.555557 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:47:24.556813 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.252:37505
--webserver_interface=127.15.102.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab
--principal=kudu/127.15.102.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:37963,127.15.102.253:38359,127.15.102.252:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:24.556902 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
I20260620 18:47:24.606341 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 1
I20260620 18:47:24.607882 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.607183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52550)
0620 18:47:24.607482 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.607638 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 18:47:24.609181 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.608734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52564)
0620 18:47:24.608919 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.609016 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:47:24.609406 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
I20260620 18:47:24.663193 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } attempt: 5
Jun 20 18:47:24 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:47:24.677642 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 2
I20260620 18:47:24.678784 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.678275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52576)
0620 18:47:24.678488 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.678588 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:47:24.679201 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.664118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50724)
0620 18:47:24.664366 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.664391 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:24.664528 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:24.664900 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:24.664903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:24.665293 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:24.665479 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:24.665489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.666966 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.666970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:24.667988 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:24.668003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.669142 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.669147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:24.669304 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:24.669976 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:24.669996 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:24.672280 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:24.676374 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:24.676378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:24.676381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:24.676804 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:24.677262 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:24.677265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:24.677268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:24.677404 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:24.678114 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:24.678121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:24.678317 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 18:47:24.678477 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 18:47:24.678830 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:47:24.679529 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.664312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.15.102.253:38359)
0620 18:47:24.664537 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:24.664543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:24.664570 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:47:24.664640 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:24.664646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:24.664745 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:24.664862 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:24.665702 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.666803 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.668176 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:24.668958 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:24.669118 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:24.672585 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:24.672595 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:24.672602 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:24.672660 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:24.676156 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:24.676954 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:24.676960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:24.676963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:24.677138 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:24.677514 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:24.677519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:24.677522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:24.677924 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:24.678136 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:24.678604 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 18:47:24.678840 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:47:24.680335 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.679889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52588)
0620 18:47:24.680081 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.680171 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:47:24.680699 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
I20260620 18:47:24.680681 29263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:24.681742 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.681359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52600)
0620 18:47:24.681524 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.681607 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:47:24.682740 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.682378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52602)
0620 18:47:24.682527 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.682612 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:47:24.682952 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
I20260620 18:47:24.734253 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 1
I20260620 18:47:24.735651 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.735077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52614)
0620 18:47:24.735342 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.735455 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 18:47:24.736745 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.736371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52620)
0620 18:47:24.736534 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.736628 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 18:47:24.736981 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
I20260620 18:47:24.771977 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 3
I20260620 18:47:24.773557 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.772932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52634)
0620 18:47:24.773188 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.773326 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:47:24.775051 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.774478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52636)
0620 18:47:24.774763 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.774902 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260620 18:47:24.775341 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
I20260620 18:47:24.805311 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 2
I20260620 18:47:24.806638 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.806105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52650)
0620 18:47:24.806339 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.806458 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 18:47:24.807787 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.807391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52664)
0620 18:47:24.807589 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.807665 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 18:47:24.808040 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
I20260620 18:47:24.914276 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 3
I20260620 18:47:24.915571 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.914994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52678)
0620 18:47:24.915220 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.915369 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:47:24.917196 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.916556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52682)
0620 18:47:24.916861 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.917028 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 18:47:24.917519 29263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
I20260620 18:47:24.918762 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 4
I20260620 18:47:24.919886 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.919406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52684)
0620 18:47:24.919632 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.919726 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:47:24.920902 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:24.920542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52696)
0620 18:47:24.920708 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:24.920789 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:47:24.921123 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:37505: Network error: Client connection negotiation failed: client connection to 127.15.102.252:37505: connect: Connection refused (error 111)
W20260620 18:47:24.922408 29345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:24.922909 29345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:24.923074 29345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:24.930970 29345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:24.931092 29345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:24.931196 29345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:24.931308 29345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:24.931375 29345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:24.942183 29345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:37963,127.15.102.253:38359,127.15.102.252:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab
--principal=kudu/127.15.102.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.252:37505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.102.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:24.943789 29345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:24.945287 29345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:24.958950 29353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:24.958921 29351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:24.959262 29350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:24.959759 29345 server_base.cc:1034] running on GCE node
I20260620 18:47:24.960672 29345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:24.962280 29345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:24.963531 29345 hybrid_clock.cc:648] HybridClock initialized: now 1781981244963485 us; error 39 us; skew 500 ppm
I20260620 18:47:24.963930 29345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:24 dist-test-slave-zfp8 krb5kdc[29183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:24.967500 29345 init.cc:377] Logged in from keytab as kudu/127.15.102.252@KRBTEST.COM (short username kudu)
I20260620 18:47:24.969095 29345 webserver.cc:460] Webserver started at http://127.15.102.252:43983/ using document root <none> and password file <none>
I20260620 18:47:24.969899 29345 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:24.970032 29345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:24.970316 29345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:24.973600 29345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/data/instance:
uuid: "556a4441fa6a4187873f13a5777e9639"
format_stamp: "Formatted at 2026-06-20 18:47:24 on dist-test-slave-zfp8"
server_key: "4588ff2957fa28eb4fc5f3b5378e0cbc"
server_key_iv: "b123be357b8f34b40a34b5513b7e6d63"
server_key_version: "encryptionkey@0"
I20260620 18:47:24.974500 29345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/wal/instance:
uuid: "556a4441fa6a4187873f13a5777e9639"
format_stamp: "Formatted at 2026-06-20 18:47:24 on dist-test-slave-zfp8"
server_key: "4588ff2957fa28eb4fc5f3b5378e0cbc"
server_key_iv: "b123be357b8f34b40a34b5513b7e6d63"
server_key_version: "encryptionkey@0"
I20260620 18:47:24.981208 29345 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260620 18:47:24.985321 29360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:24.986915 29345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:47:24.987136 29345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/wal
uuid: "556a4441fa6a4187873f13a5777e9639"
format_stamp: "Formatted at 2026-06-20 18:47:24 on dist-test-slave-zfp8"
server_key: "4588ff2957fa28eb4fc5f3b5378e0cbc"
server_key_iv: "b123be357b8f34b40a34b5513b7e6d63"
server_key_version: "encryptionkey@0"
I20260620 18:47:24.987442 29345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:25.014160 29345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:25.019100 29345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:25.019481 29345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:25.039436 29345 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.252:37505
I20260620 18:47:25.039439 29411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.252:37505 every 8 connection(s)
I20260620 18:47:25.041020 29345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
I20260620 18:47:25.041551 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 29345
I20260620 18:47:25.041775 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781981043007005-15771-0/minicluster-data/master-2/wal/instance
I20260620 18:47:25.042459 15771 external_mini_cluster.cc:1428] Setting key 6fa2d5037dd002c165efd99f1da42696
I20260620 18:47:25.046165 29412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:25.055447 29412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:25 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
Jun 20 18:47:25 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:25.069132 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.044734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.15.102.252:37505)
0620 18:47:25.045598 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.045610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.045657 (+ 47us) server_negotiation.cc:407] Connection header received
0620 18:47:25.047506 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.047532 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.047961 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.048326 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.050353 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.051890 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.054038 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.054771 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.054914 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.060251 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.060285 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.060308 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.060363 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.064088 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.065102 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.065112 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.065122 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.065348 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.065760 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.065767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.065773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.066458 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.066790 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.067406 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.067891 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":389,"threads_started":1}
I20260620 18:47:25.076450 29416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.058985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.15.102.254:37963)
0620 18:47:25.059585 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.059593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.060188 (+ 595us) server_negotiation.cc:407] Connection header received
0620 18:47:25.060347 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.060352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.060439 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.060562 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.061975 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.063146 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.065098 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.065714 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.065861 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.069349 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.069356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.069363 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.069427 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.073026 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.074082 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.074090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.074094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.074257 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.074655 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.074660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.074664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.074989 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.075173 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.075726 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.075978 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":312,"threads_started":1}
I20260620 18:47:25.076439 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.058971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:37963 (local address 127.0.0.1:33804)
0620 18:47:25.059976 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:25.060027 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:25.060214 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:25.060612 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:25.060618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:25.061459 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:25.061795 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.061809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.063381 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.063385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.064875 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.064893 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.065917 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.065922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.066055 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.066764 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:25.066781 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:25.069094 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:25.073244 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.073258 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.073265 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.073950 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.074387 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.074394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.074397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.074539 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.075145 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:25.075154 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:25.075405 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 18:47:25.075601 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 18:47:25.075969 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":596,"threads_started":1}
I20260620 18:47:25.078038 29412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:47:25 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
Jun 20 18:47:25 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:47:25.090935 29418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.073215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.15.102.254:37963)
0620 18:47:25.073935 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.073939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.073959 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:47:25.074004 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.074009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.074066 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.074193 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.075694 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.076821 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.078313 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.078857 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.078997 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.083664 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.083674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.083681 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.083730 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.087624 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.088645 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.088649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.088651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.088794 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.089287 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.089291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.089294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.089675 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.089988 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.090253 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.090497 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":319,"threads_started":1}
I20260620 18:47:25.092741 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.078940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.15.102.253:38359)
0620 18:47:25.079133 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.079140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.079160 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:47:25.079224 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.079227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.079358 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.079463 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.080186 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.080894 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.082202 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.082741 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.082844 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.086330 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.086335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.086339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.086376 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.089506 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.090420 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.090427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.090430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.090619 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.091172 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.091177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.091181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.091561 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.091740 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.092133 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.092348 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:47:25.092869 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.078666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:38359 (local address 127.0.0.1:50730)
0620 18:47:25.078943 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:25.078963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:25.079065 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:25.079489 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:25.079494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:25.079876 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:25.080027 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.080035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.081032 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.081039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.082032 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.082046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.082926 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.082932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.083174 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.083856 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:25.083874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:25.086109 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:25.089663 (+ 3554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.089669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.089672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.090275 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.090758 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.090764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.090768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.091003 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.091722 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:25.091726 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:25.091867 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 18:47:25.092030 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 18:47:25.092338 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 18:47:25.093954 29412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:47:25.094815 29263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 4
Jun 20 18:47:25 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
Jun 20 18:47:25 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
Jun 20 18:47:25 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981243, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
I20260620 18:47:25.111163 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.094639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52718)
0620 18:47:25.094830 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:25.094852 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:25.094952 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:25.095495 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:25.095501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:25.096016 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:25.096221 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.096232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.097604 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.097613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.098976 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.098990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.099923 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.099929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.100108 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.100766 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:25.100795 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:25.103135 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:25.107374 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.107384 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.107392 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.108403 (+ 1011us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.108895 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.108903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.108909 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.109106 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.109969 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:25.109973 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:25.110177 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 18:47:25.110355 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 18:47:25.110717 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 18:47:25.111974 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.095480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52730)
0620 18:47:25.095791 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:25.095815 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:25.095944 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:25.096880 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:25.096886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:25.097309 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:25.097476 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.097486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.098738 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.098744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.100034 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.100053 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.101054 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.101063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.101277 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.101949 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:25.101978 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:25.105427 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:25.109438 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.109444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.109448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.109871 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.110278 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.110283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.110287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.110440 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.110988 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:25.110993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:25.111147 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 18:47:25.111340 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 18:47:25.111683 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 18:47:25.112061 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.094722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.15.102.252:37505)
0620 18:47:25.094891 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.094897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.094915 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:47:25.095091 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.095096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.095181 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.095305 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.096381 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.097457 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.099136 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.099755 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.099903 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.103363 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.103370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.103376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.103436 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.107179 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.108563 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.108568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.108571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.108741 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.109237 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.109245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.109251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.109763 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.109905 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.111501 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.111739 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:47:25.112824 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.093100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.15.102.253:38359)
0620 18:47:25.093271 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.093276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.093449 (+ 173us) server_negotiation.cc:407] Connection header received
0620 18:47:25.093604 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.093609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.093684 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.093803 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.095445 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.096682 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.098595 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.099303 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.099407 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.106093 (+ 6686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.106103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.106109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.106157 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.109141 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.110154 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.110160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.110163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.110378 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.110882 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.110889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.110893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.111310 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.111592 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.112214 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.112465 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:47:25.113005 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.095517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.15.102.252:37505)
0620 18:47:25.096573 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.096577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.096597 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:47:25.096635 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.096639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.096738 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.096839 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.097659 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.098596 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.100183 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.100871 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.100990 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.105703 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.105709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.105712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.105747 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.109283 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.109999 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.110013 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.110016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.110152 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.110570 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.110581 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.110584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.110851 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.111010 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.112444 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.112643 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":948,"thread_start_us":740,"threads_started":1}
I20260620 18:47:25.117604 29412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639: Bootstrap starting.
I20260620 18:47:25.118192 29263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: Bootstrap starting.
I20260620 18:47:25.120723 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:25.121798 29263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:25.121848 29412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:25.122900 29263 log.cc:824] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:25.123111 29412 log.cc:824] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:25.126472 29263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: No bootstrap required, opened a new log
I20260620 18:47:25.131219 29412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639: No bootstrap required, opened a new log
I20260620 18:47:25.143275 29263 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
I20260620 18:47:25.143781 29263 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:25.143930 29263 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf454adc8bd74cd7bc6fba734628ed6e, State: Initialized, Role: FOLLOWER
I20260620 18:47:25.144620 29412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
I20260620 18:47:25.144876 29263 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
I20260620 18:47:25.145293 29412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:25.145478 29412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556a4441fa6a4187873f13a5777e9639, State: Initialized, Role: FOLLOWER
I20260620 18:47:25.146430 29412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
I20260620 18:47:25.149231 29418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.128607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.15.102.254:37963)
0620 18:47:25.128814 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.128820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.128846 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:47:25.129065 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.129071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.129154 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.129238 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.131778 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.132973 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.135382 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.136150 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.136266 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.140723 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.140731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.140737 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.140789 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.144730 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.145879 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.145886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.145889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.146083 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.146653 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.146658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.146662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.147070 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.147458 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.148255 (+ 797us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.148659 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:47:25.166383 29429 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.166957 29429 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:25.167650 29430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.168020 29263 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:25.168104 29430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:25.169379 29412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:25.171286 29342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.129760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.15.102.253:38359)
0620 18:47:25.129942 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.129948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.139959 (+ 10011us) server_negotiation.cc:407] Connection header received
0620 18:47:25.140328 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.140335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.140432 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.140511 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.142711 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.143912 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.146866 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.147894 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.148036 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.157864 (+ 9828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.157877 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.157886 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.157963 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.163068 (+ 5105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.167990 (+ 4922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.167999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.168004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.168234 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.168787 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.168792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.168795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.169213 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.169494 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.170314 (+ 820us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.170673 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:47:25.177376 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.140280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.15.102.252:37505)
0620 18:47:25.140546 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.140552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.150701 (+ 10149us) server_negotiation.cc:407] Connection header received
0620 18:47:25.151066 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.151075 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.151197 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.151337 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.155200 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.156782 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.159960 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.161017 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.161172 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.168389 (+ 7217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.168396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.168400 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.168443 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.172698 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.173891 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.173901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.173907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.174110 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.174726 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.174735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.174741 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.175289 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.175640 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.176454 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.176845 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260620 18:47:25.181852 29217 tablet.cc:2369] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:47:25.193454 29450 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:47:25.193641 29450 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 18:47:25.200872 29452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:47:25.201038 29452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:47:25.195469 29196 master_runner.cc:186] Error getting master registration for 127.15.102.253:38359: OK, instance_id { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" instance_seqno: 1781981244481577 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:47:25.210799 29196 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:47:25.191816 29345 master_runner.cc:186] Error getting master registration for 127.15.102.253:38359: OK, instance_id { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" instance_seqno: 1781981244481577 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:47:25.212754 29345 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:47:25.220373 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } attempt: 5
Jun 20 18:47:25 dist-test-slave-zfp8 krb5kdc[29183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
I20260620 18:47:25.239562 29340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.221039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:37505 (local address 127.0.0.1:52746)
0620 18:47:25.221308 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:25.221330 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:25.221461 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:25.221871 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:25.221878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:25.222430 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:25.222658 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.222674 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.224337 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.224343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.225797 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:25.225813 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.227890 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.227897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:25.228116 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.228863 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:25.228890 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:25.231857 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:25.236293 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.236300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.236305 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.236862 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.237420 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:25.237425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:25.237429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:25.237649 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:25.238439 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:25.238444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:25.238636 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 18:47:25.238779 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 18:47:25.239113 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 18:47:25.240146 29415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:25.221126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.15.102.252:37505)
0620 18:47:25.221311 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:25.221317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:25.221423 (+ 106us) server_negotiation.cc:407] Connection header received
0620 18:47:25.221608 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:25.221614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:25.221701 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:25.221820 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:25.222848 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.224151 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.226931 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:25.227709 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:25.227851 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:25.232107 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:25.232115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:25.232120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:25.232164 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:25.236094 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.237023 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.237030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.237033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.237216 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:25.237792 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:25.237802 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:25.237806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:25.238194 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:25.239302 (+ 1108us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:25.239452 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:47:25.239724 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:47:25.246485 29337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6: Bootstrap starting.
I20260620 18:47:25.251134 29337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:25.252638 29337 log.cc:824] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:25.256531 29337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6: No bootstrap required, opened a new log
I20260620 18:47:25.257702 29430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:47:25.258041 29430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
I20260620 18:47:25.259887 29430 leader_election.cc:288] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf454adc8bd74cd7bc6fba734628ed6e (127.15.102.254:37963), 4054b15c16274bcd9b31752e95da9cf6 (127.15.102.253:38359)
I20260620 18:47:25.264104 29238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "556a4441fa6a4187873f13a5777e9639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" is_pre_election: true
I20260620 18:47:25.264796 29238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556a4441fa6a4187873f13a5777e9639 in term 0.
I20260620 18:47:25.265724 29361 leader_election.cc:302] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556a4441fa6a4187873f13a5777e9639, bf454adc8bd74cd7bc6fba734628ed6e; no voters:
I20260620 18:47:25.268324 29430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:47:25.268520 29430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:47:25.268632 29430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:25.268617 29337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
I20260620 18:47:25.269130 29337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:25.269306 29337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4054b15c16274bcd9b31752e95da9cf6, State: Initialized, Role: FOLLOWER
I20260620 18:47:25.270757 29337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
W20260620 18:47:25.271886 29366 tablet.cc:2369] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:25.272207 29430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
I20260620 18:47:25.273814 29238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "556a4441fa6a4187873f13a5777e9639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf454adc8bd74cd7bc6fba734628ed6e"
I20260620 18:47:25.274081 29238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:25.278841 29238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556a4441fa6a4187873f13a5777e9639 in term 1.
I20260620 18:47:25.279762 29361 leader_election.cc:302] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556a4441fa6a4187873f13a5777e9639, bf454adc8bd74cd7bc6fba734628ed6e; no voters:
I20260620 18:47:25.281272 29312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "556a4441fa6a4187873f13a5777e9639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4054b15c16274bcd9b31752e95da9cf6" is_pre_election: true
I20260620 18:47:25.281248 29337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:25.284065 29312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556a4441fa6a4187873f13a5777e9639 in term 0.
I20260620 18:47:25.285537 29456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.285995 29456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:25.287310 29430 leader_election.cc:288] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [CANDIDATE]: Term 1 election: Requested vote from peers bf454adc8bd74cd7bc6fba734628ed6e (127.15.102.254:37963), 4054b15c16274bcd9b31752e95da9cf6 (127.15.102.253:38359)
I20260620 18:47:25.287612 29430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:25.287782 29430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [term 1 LEADER]: Becoming Leader. State: Replica: 556a4441fa6a4187873f13a5777e9639, State: Running, Role: LEADER
I20260620 18:47:25.288436 29430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } }
I20260620 18:47:25.302148 29312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "556a4441fa6a4187873f13a5777e9639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4054b15c16274bcd9b31752e95da9cf6"
I20260620 18:47:25.302465 29312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:25.306469 29312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556a4441fa6a4187873f13a5777e9639 in term 1.
I20260620 18:47:25.313519 29457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556a4441fa6a4187873f13a5777e9639. Latest consensus state: current_term: 1 leader_uuid: "556a4441fa6a4187873f13a5777e9639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.314118 29457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:25.321213 29464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:25.323985 29464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:25.346262 29312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [term 1 FOLLOWER]: Refusing update from remote peer 556a4441fa6a4187873f13a5777e9639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:25.350327 29457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:25.359215 29238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [term 1 FOLLOWER]: Refusing update from remote peer 556a4441fa6a4187873f13a5777e9639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:25.360507 29457 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:25.378310 29456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556a4441fa6a4187873f13a5777e9639. Latest consensus state: current_term: 1 leader_uuid: "556a4441fa6a4187873f13a5777e9639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.378744 29456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:25.390733 29429 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556a4441fa6a4187873f13a5777e9639. Latest consensus state: current_term: 1 leader_uuid: "556a4441fa6a4187873f13a5777e9639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.391081 29429 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:25.392261 29468 mvcc.cc:205] Tried to move back new op lower bound from 7298995180920463360 to 7298995180720377856. Current Snapshot: MvccSnapshot[applied={T|T < 7298995180920463360}]
I20260620 18:47:25.398896 29472 mvcc.cc:205] Tried to move back new op lower bound from 7298995180920463360 to 7298995180720377856. Current Snapshot: MvccSnapshot[applied={T|T < 7298995180920463360}]
I20260620 18:47:25.406692 29430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "556a4441fa6a4187873f13a5777e9639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.407141 29430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:25.407538 29430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "556a4441fa6a4187873f13a5777e9639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.407819 29430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:25.413478 29429 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "556a4441fa6a4187873f13a5777e9639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.413838 29429 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:25.415882 29456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "556a4441fa6a4187873f13a5777e9639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf454adc8bd74cd7bc6fba734628ed6e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37963 } } peers { permanent_uuid: "4054b15c16274bcd9b31752e95da9cf6" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 38359 } } peers { permanent_uuid: "556a4441fa6a4187873f13a5777e9639" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 37505 } } }
I20260620 18:47:25.416153 29456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:47:25.424741 29464 catalog_manager.cc:1347] Generated new cluster ID: b623e470860c433da27c4af098da5d6e
I20260620 18:47:25.424880 29464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260620 18:47:25.425534 29475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:47:25.425693 29475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:47:25.468026 29464 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:25.470121 29464 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:25.496232 29464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556a4441fa6a4187873f13a5777e9639: Generated new TSK 0
I20260620 18:47:25.497465 29464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:26.195999 29450 catalog_manager.cc:1259] Loaded cluster ID: b623e470860c433da27c4af098da5d6e
I20260620 18:47:26.196110 29450 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: loading cluster ID for follower catalog manager: success
I20260620 18:47:26.198714 29450 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: acquiring CA information for follower catalog manager: success
I20260620 18:47:26.200826 29450 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf454adc8bd74cd7bc6fba734628ed6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:47:26.427714 29475 catalog_manager.cc:1259] Loaded cluster ID: b623e470860c433da27c4af098da5d6e
I20260620 18:47:26.427871 29475 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6: loading cluster ID for follower catalog manager: success
I20260620 18:47:26.430491 29475 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6: acquiring CA information for follower catalog manager: success
I20260620 18:47:26.432626 29475 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4054b15c16274bcd9b31752e95da9cf6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:47:26.622778 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 29196
I20260620 18:47:26.624229 29258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:26.737416 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 29196
I20260620 18:47:26.759963 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 29270
I20260620 18:47:26.761268 29332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:26.869241 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 29270
I20260620 18:47:26.891031 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 29345
I20260620 18:47:26.892530 29407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:27.008431 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 29345
2026-06-20T18:47:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4878 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:27 dist-test-slave-zfp8 krb5kdc[29490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:27 dist-test-slave-zfp8 krb5kdc[29490](info): set up 2 sockets
Jun 20 18:47:27 dist-test-slave-zfp8 krb5kdc[29490](info): commencing operation
krb5kdc: starting...
W20260620 18:47:28.131944 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:47:28 dist-test-slave-zfp8 krb5kdc[29490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:47:28.370345 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35697
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:35697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:28.728243 29503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:28.728664 29503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:28.728839 29503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:28.736466 29503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:28.736665 29503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:28.736788 29503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:28.736907 29503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:28.736997 29503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:28.747426 29503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:35697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:28.748934 29503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:28.750378 29503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:28.763391 29508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:28.763481 29511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:28.763496 29509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:28.764289 29503 server_base.cc:1034] running on GCE node
I20260620 18:47:28.765304 29503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:28.766995 29503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:28.768246 29503 hybrid_clock.cc:648] HybridClock initialized: now 1781981248768181 us; error 57 us; skew 500 ppm
I20260620 18:47:28.768635 29503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:28 dist-test-slave-zfp8 krb5kdc[29490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:28.772030 29503 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:47:28.773772 29503 webserver.cc:460] Webserver started at http://127.15.102.254:42557/ using document root <none> and password file <none>
I20260620 18:47:28.774556 29503 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:28.774713 29503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:28.775069 29503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:28.778363 29503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "60930f1b41f94971bd7169845ffbd015"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-zfp8"
server_key: "06ed4dd073922208c393c8eeaf0f3a93"
server_key_iv: "1dfa129365132eac724e4817bbde181e"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.779285 29503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "60930f1b41f94971bd7169845ffbd015"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-zfp8"
server_key: "06ed4dd073922208c393c8eeaf0f3a93"
server_key_iv: "1dfa129365132eac724e4817bbde181e"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.785106 29503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:47:28.789075 29518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:28.790659 29503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:47:28.790876 29503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "60930f1b41f94971bd7169845ffbd015"
format_stamp: "Formatted at 2026-06-20 18:47:28 on dist-test-slave-zfp8"
server_key: "06ed4dd073922208c393c8eeaf0f3a93"
server_key_iv: "1dfa129365132eac724e4817bbde181e"
server_key_version: "encryptionkey@0"
I20260620 18:47:28.791107 29503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:28.807761 29503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:28.812023 29503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:28.812354 29503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:28.831615 29503 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:35697
I20260620 18:47:28.831670 29569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:35697 every 8 connection(s)
I20260620 18:47:28.833133 29503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:28.835561 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 29503
I20260620 18:47:28.835811 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:47:28.836480 15771 external_mini_cluster.cc:1428] Setting key 2cc767fa59b80822e9b9e2c4852510b9
I20260620 18:47:28.837468 29570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:28.850111 29570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015: Bootstrap starting.
Jun 20 18:47:28 dist-test-slave-zfp8 krb5kdc[29490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:28.853665 29570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:28.854949 29570 log.cc:824] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:28.858749 29570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015: No bootstrap required, opened a new log
I20260620 18:47:28.860863 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:28.838436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.15.102.254:35697)
0620 18:47:28.839287 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:28.839308 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:28.839359 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:47:28.841066 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:28.841089 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:28.841522 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:28.841958 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:28.843629 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.844986 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.846428 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:28.847161 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:28.847413 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:28.852204 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:28.852214 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:28.852230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:28.852289 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:28.856279 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.857279 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.857294 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.857301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.857501 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:28.857953 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:28.857957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:28.857959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:28.858557 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:28.858784 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:28.859374 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 18:47:28.859780 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":343,"threads_started":1}
I20260620 18:47:28.865039 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:47:28.868429 29570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60930f1b41f94971bd7169845ffbd015" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35697 } }
I20260620 18:47:28.868876 29570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:28.869010 29570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60930f1b41f94971bd7169845ffbd015, State: Initialized, Role: FOLLOWER
I20260620 18:47:28.869696 29570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60930f1b41f94971bd7169845ffbd015" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35697 } }
I20260620 18:47:28.869925 29570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:28.870080 29570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:28.870265 29570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:28.872681 29570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60930f1b41f94971bd7169845ffbd015" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35697 } }
I20260620 18:47:28.873255 29570 leader_election.cc:302] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60930f1b41f94971bd7169845ffbd015; no voters:
I20260620 18:47:28.873891 29570 leader_election.cc:288] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:28.874060 29575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:28.874781 29575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [term 1 LEADER]: Becoming Leader. State: Replica: 60930f1b41f94971bd7169845ffbd015, State: Running, Role: LEADER
I20260620 18:47:28.875373 29575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60930f1b41f94971bd7169845ffbd015" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35697 } }
I20260620 18:47:28.876191 29570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:28.878510 29577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60930f1b41f94971bd7169845ffbd015. Latest consensus state: current_term: 1 leader_uuid: "60930f1b41f94971bd7169845ffbd015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60930f1b41f94971bd7169845ffbd015" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35697 } } }
I20260620 18:47:28.878885 29577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:28.878607 29576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60930f1b41f94971bd7169845ffbd015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60930f1b41f94971bd7169845ffbd015" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35697 } } }
I20260620 18:47:28.879207 29576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:28.880959 29581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:28.885716 29581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:28.895709 29581 catalog_manager.cc:1347] Generated new cluster ID: b3284f5f2d204dccbada72980560b691
I20260620 18:47:28.895865 29581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:28.909310 29581 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:28.910751 29581 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:28.927362 29581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60930f1b41f94971bd7169845ffbd015: Generated new TSK 0
I20260620 18:47:28.928361 29581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:47:28.979612 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 29503
I20260620 18:47:28.980984 29565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:29.079658 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 29503
2026-06-20T18:47:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2065 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:29 dist-test-slave-zfp8 krb5kdc[29598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:29 dist-test-slave-zfp8 krb5kdc[29598](info): set up 2 sockets
Jun 20 18:47:29 dist-test-slave-zfp8 krb5kdc[29598](info): commencing operation
krb5kdc: starting...
W20260620 18:47:30.183848 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:47:30 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:47:30.432909 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:34189
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:30.791694 29611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:30.792042 29611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:30.792222 29611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:30.800088 29611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:30.800223 29611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:30.800352 29611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:30.800442 29611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:30.800511 29611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:30.811101 29611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:30.812731 29611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:30.814199 29611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:30.827201 29617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:30.827293 29616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:30.827273 29619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:30.828617 29611 server_base.cc:1034] running on GCE node
I20260620 18:47:30.829545 29611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:30.831161 29611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:30.832453 29611 hybrid_clock.cc:648] HybridClock initialized: now 1781981250832404 us; error 47 us; skew 500 ppm
I20260620 18:47:30.832859 29611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:30 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981250, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:30.836503 29611 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:47:30.838254 29611 webserver.cc:460] Webserver started at http://127.15.102.254:34503/ using document root <none> and password file <none>
I20260620 18:47:30.839049 29611 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:30.839212 29611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:30.839612 29611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:30.842891 29611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "394f633ec05c4078a6a98e8273b12c70"
format_stamp: "Formatted at 2026-06-20 18:47:30 on dist-test-slave-zfp8"
server_key: "81e4d7cb58f21d8fa9bbf3767bc2b39c"
server_key_iv: "07e4583bb4a247dc42f54a856ff27fd5"
server_key_version: "encryptionkey@0"
I20260620 18:47:30.843794 29611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "394f633ec05c4078a6a98e8273b12c70"
format_stamp: "Formatted at 2026-06-20 18:47:30 on dist-test-slave-zfp8"
server_key: "81e4d7cb58f21d8fa9bbf3767bc2b39c"
server_key_iv: "07e4583bb4a247dc42f54a856ff27fd5"
server_key_version: "encryptionkey@0"
I20260620 18:47:30.849828 29611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 18:47:30.853864 29626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:30.855520 29611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:47:30.855742 29611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "394f633ec05c4078a6a98e8273b12c70"
format_stamp: "Formatted at 2026-06-20 18:47:30 on dist-test-slave-zfp8"
server_key: "81e4d7cb58f21d8fa9bbf3767bc2b39c"
server_key_iv: "07e4583bb4a247dc42f54a856ff27fd5"
server_key_version: "encryptionkey@0"
I20260620 18:47:30.855980 29611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:30.869906 29611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:30.874749 29611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:30.875093 29611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:30.894822 29611 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:34189
I20260620 18:47:30.894845 29677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:34189 every 8 connection(s)
I20260620 18:47:30.896472 29611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:30.898607 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 29611
I20260620 18:47:30.898864 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:47:30.899708 15771 external_mini_cluster.cc:1428] Setting key abcefde172d837a58391d95c51e899b6
I20260620 18:47:30.901014 29678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:47:30 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:30.916826 29678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Bootstrap starting.
I20260620 18:47:30.920468 29678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:30.921609 29678 log.cc:824] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:30.924938 29678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: No bootstrap required, opened a new log
I20260620 18:47:30.924940 29681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:30.901960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.15.102.254:34189)
0620 18:47:30.902825 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:30.902838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:30.902883 (+ 45us) server_negotiation.cc:407] Connection header received
0620 18:47:30.904606 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:30.904634 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:30.905124 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:30.905508 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:30.907516 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.908963 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.910851 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:30.911665 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:30.911819 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:30.916936 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:30.916945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:30.916963 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:30.917054 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:30.920404 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:30.921232 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:30.921239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:30.921247 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:30.921439 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:30.921888 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:30.921893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:30.921898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:30.922512 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:30.922883 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:30.923384 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 18:47:30.923856 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":401,"threads_started":1}
I20260620 18:47:30.933724 29678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:30.934247 29678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:30.934386 29678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394f633ec05c4078a6a98e8273b12c70, State: Initialized, Role: FOLLOWER
I20260620 18:47:30.935196 29678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:30.935544 29678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:30.935719 29678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:30.935907 29678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:30.938858 29678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:30.939533 29678 leader_election.cc:302] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 394f633ec05c4078a6a98e8273b12c70; no voters:
I20260620 18:47:30.940491 29678 leader_election.cc:288] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:30.940661 29683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:30.943457 29678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:30.944211 29683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 1 LEADER]: Becoming Leader. State: Replica: 394f633ec05c4078a6a98e8273b12c70, State: Running, Role: LEADER
I20260620 18:47:30.945025 29683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:30.948403 29684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "394f633ec05c4078a6a98e8273b12c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } } }
I20260620 18:47:30.948818 29684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:30.948516 29685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394f633ec05c4078a6a98e8273b12c70. Latest consensus state: current_term: 1 leader_uuid: "394f633ec05c4078a6a98e8273b12c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } } }
I20260620 18:47:30.948966 29685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:30.952205 29693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:30.956859 29693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:30.967571 29693 catalog_manager.cc:1347] Generated new cluster ID: 557950285866474297c662a6d5afbd56
I20260620 18:47:30.967759 29693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:31.017076 29693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:31.018582 29693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:31.029069 29693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Generated new TSK 0
I20260620 18:47:31.030251 29693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:47:31.096818 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34189
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:31.470623 29704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:31.471012 29704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:31.471114 29704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:31.478672 29704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:31.478822 29704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:31.478947 29704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:47:31.489442 29704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:31.490942 29704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:31.492497 29704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:31.506867 29709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:31.506978 29710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:31.507023 29712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:31.508117 29704 server_base.cc:1034] running on GCE node
I20260620 18:47:31.508868 29704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:31.509994 29704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:31.511329 29704 hybrid_clock.cc:648] HybridClock initialized: now 1781981251511203 us; error 62 us; skew 500 ppm
I20260620 18:47:31.511721 29704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:31 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981251, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:31.515183 29704 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:47:31.517007 29704 webserver.cc:460] Webserver started at http://127.15.102.193:35453/ using document root <none> and password file <none>
I20260620 18:47:31.517758 29704 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:31.517932 29704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:31.518221 29704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:31.521489 29704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "88dd7880435b43c39f624069b8025012"
format_stamp: "Formatted at 2026-06-20 18:47:31 on dist-test-slave-zfp8"
server_key: "a85d9409411090332545321e7169c8a8"
server_key_iv: "7e530f3589941b6a51842cb0f1fdf149"
server_key_version: "encryptionkey@0"
I20260620 18:47:31.522337 29704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "88dd7880435b43c39f624069b8025012"
format_stamp: "Formatted at 2026-06-20 18:47:31 on dist-test-slave-zfp8"
server_key: "a85d9409411090332545321e7169c8a8"
server_key_iv: "7e530f3589941b6a51842cb0f1fdf149"
server_key_version: "encryptionkey@0"
I20260620 18:47:31.528268 29704 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 18:47:31.532444 29719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:31.534138 29704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260620 18:47:31.534319 29704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "88dd7880435b43c39f624069b8025012"
format_stamp: "Formatted at 2026-06-20 18:47:31 on dist-test-slave-zfp8"
server_key: "a85d9409411090332545321e7169c8a8"
server_key_iv: "7e530f3589941b6a51842cb0f1fdf149"
server_key_version: "encryptionkey@0"
I20260620 18:47:31.534543 29704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:31.557101 29704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:31.561834 29704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:31.562105 29704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:31.563172 29704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:31.564678 29704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:31.564764 29704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:31.564857 29704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:31.564905 29704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:31.600718 29704 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:41367
I20260620 18:47:31.600764 29831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:41367 every 8 connection(s)
I20260620 18:47:31.602262 29704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:47:31.603704 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 29704
I20260620 18:47:31.603963 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:47:31.604826 15771 external_mini_cluster.cc:1428] Setting key 8277be236b3aba190f6f18345b43e282
Jun 20 18:47:31 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981251, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:31.627744 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:31.605481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:32837 (local address 127.15.102.254:34189)
0620 18:47:31.605997 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:31.606004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:31.607326 (+ 1322us) server_negotiation.cc:407] Connection header received
0620 18:47:31.609330 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:31.609335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:31.609453 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:31.609581 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:31.611343 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:31.612126 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:31.614248 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:31.614685 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:31.614806 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:31.619892 (+ 5086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:31.619902 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:31.619909 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:31.619965 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:31.623857 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:31.624803 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:31.624812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:31.624818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:31.624988 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:31.625448 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:31.625456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:31.625462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:31.625910 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:31.626078 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:31.626928 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 18:47:31.627226 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":268,"threads_started":1}
I20260620 18:47:31.628751 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:31.606187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:32837)
0620 18:47:31.607130 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:31.607174 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:31.609015 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:31.609867 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:31.609876 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:31.610474 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:31.611110 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:31.611135 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:31.612326 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:31.612331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:31.614021 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:31.614040 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:31.614969 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:31.614979 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:31.615384 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:31.616215 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:31.616252 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:31.619568 (+ 3316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:31.624076 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:31.624085 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:31.624102 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:31.624621 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:31.625110 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:31.625114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:31.625116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:31.625312 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:31.626078 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:31.626085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:31.626481 (+ 396us) client_negotiation.cc:769] Sending connection context
0620 18:47:31.626735 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 18:47:31.627197 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:47:31.630725 29832 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34189
I20260620 18:47:31.631076 29832 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:31.631949 29832 heartbeater.cc:511] Master 127.15.102.254:34189 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
I20260620 18:47:31.634428 29643 ts_manager.cc:194] Registered new tserver with Master: 88dd7880435b43c39f624069b8025012 (127.15.102.193:41367)
I20260620 18:47:31.636070 29643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:32837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:47:31.662870 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34189
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:32.023979 29838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:32.024277 29838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:32.024442 29838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:32.032253 29838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:32.032374 29838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:32.032564 29838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:47:32.043016 29838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:32.044618 29838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:32.046137 29838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:32.061261 29843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:32.061290 29844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:32.061396 29846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:32.061846 29838 server_base.cc:1034] running on GCE node
I20260620 18:47:32.062685 29838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:32.063781 29838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:32.064989 29838 hybrid_clock.cc:648] HybridClock initialized: now 1781981252064914 us; error 63 us; skew 500 ppm
I20260620 18:47:32.065322 29838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:32 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:32.068590 29838 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:47:32.070202 29838 webserver.cc:460] Webserver started at http://127.15.102.194:33611/ using document root <none> and password file <none>
I20260620 18:47:32.070932 29838 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:32.071080 29838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:32.071472 29838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:32.074497 29838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "44a2bc11bbad4c33930efb5a940c3506"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "1ba05c92acdaad88ea85d94f51a69477"
server_key_iv: "c63e6e684f637c955808bdfe9a802f95"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.075395 29838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "44a2bc11bbad4c33930efb5a940c3506"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "1ba05c92acdaad88ea85d94f51a69477"
server_key_iv: "c63e6e684f637c955808bdfe9a802f95"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.080905 29838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:47:32.084586 29853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:32.086056 29838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:47:32.086251 29838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "44a2bc11bbad4c33930efb5a940c3506"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "1ba05c92acdaad88ea85d94f51a69477"
server_key_iv: "c63e6e684f637c955808bdfe9a802f95"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.086478 29838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:32.123454 29838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:32.127740 29838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:32.128087 29838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:32.129130 29838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:32.130703 29838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:32.130815 29838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:32.130973 29838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:32.131055 29838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:32.166191 29838 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:44163
I20260620 18:47:32.166229 29965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:44163 every 8 connection(s)
I20260620 18:47:32.167649 29838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:47:32.168082 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 29838
I20260620 18:47:32.168292 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:47:32.168987 15771 external_mini_cluster.cc:1428] Setting key 318a76b886f087a2c0aff3657b8cbe5d
Jun 20 18:47:32 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:32.193423 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.170632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:35109 (local address 127.15.102.254:34189)
0620 18:47:32.171300 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:32.171307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:32.172474 (+ 1167us) server_negotiation.cc:407] Connection header received
0620 18:47:32.175574 (+ 3100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:32.175579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:32.175655 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:32.175757 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:32.178140 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.178902 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.180974 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.181764 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.181921 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.185854 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:32.185863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:32.185871 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:32.185922 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:32.189715 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:32.190642 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:32.190651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:32.190657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:32.190835 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:32.191380 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:32.191386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:32.191390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:32.191819 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:32.191996 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:32.192747 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 18:47:32.193009 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":397,"threads_started":1}
I20260620 18:47:32.194490 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.171415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:35109)
0620 18:47:32.172283 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:32.172336 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:32.175152 (+ 2816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:32.176079 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:32.176093 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:32.176993 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:32.177919 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.177949 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.179076 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.179085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.180791 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.180804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.181997 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.182002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.182224 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.182965 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:32.182990 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:32.185568 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:32.189904 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:32.189911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:32.189926 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:32.190480 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:32.190944 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:32.190947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:32.190949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:32.191202 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:32.191959 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:32.191967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:32.192366 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 18:47:32.192650 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 18:47:32.193233 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":321,"threads_started":1}
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
I20260620 18:47:32.196101 29966 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34189
I20260620 18:47:32.196488 29966 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:32.197243 29966 heartbeater.cc:511] Master 127.15.102.254:34189 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
I20260620 18:47:32.199033 29643 ts_manager.cc:194] Registered new tserver with Master: 44a2bc11bbad4c33930efb5a940c3506 (127.15.102.194:44163)
I20260620 18:47:32.199954 29643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:35109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:47:32.224962 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34189
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:32.581501 29972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:32.581830 29972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:32.581993 29972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:32.589711 29972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:32.589871 29972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:32.590137 29972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:47:32.600656 29972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:32.602036 29972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:32.603681 29972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:32.618119 29978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:32.618081 29980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:32.618042 29977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:32.619110 29972 server_base.cc:1034] running on GCE node
I20260620 18:47:32.619827 29972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:32.620865 29972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:32.622107 29972 hybrid_clock.cc:648] HybridClock initialized: now 1781981252622046 us; error 54 us; skew 500 ppm
I20260620 18:47:32.622450 29972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:32 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:32.625890 29972 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:47:32.627539 29972 webserver.cc:460] Webserver started at http://127.15.102.195:36973/ using document root <none> and password file <none>
I20260620 18:47:32.628302 29972 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:32.628468 29972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:32.628819 29972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:32.632020 29972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "86b120ef400f4152a059a9d5189e7a1c"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "ca4449eee88d5050348af4828b5ff4ee"
server_key_iv: "919363098dadf269a4f4299d7613c304"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.633173 29972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "86b120ef400f4152a059a9d5189e7a1c"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "ca4449eee88d5050348af4828b5ff4ee"
server_key_iv: "919363098dadf269a4f4299d7613c304"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.638978 29972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:47:32.639829 29832 heartbeater.cc:503] Master 127.15.102.254:34189 was elected leader, sending a full tablet report...
I20260620 18:47:32.643709 29987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:32.645630 29972 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:47:32.645859 29972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "86b120ef400f4152a059a9d5189e7a1c"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "ca4449eee88d5050348af4828b5ff4ee"
server_key_iv: "919363098dadf269a4f4299d7613c304"
server_key_version: "encryptionkey@0"
I20260620 18:47:32.646169 29972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:32.666806 29972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:32.671921 29972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:32.672261 29972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:32.673380 29972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:32.674888 29972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:32.674998 29972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:32.675143 29972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:32.675196 29972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:32.709389 29972 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:44537
I20260620 18:47:32.709443 30099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:44537 every 8 connection(s)
I20260620 18:47:32.710830 29972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:47:32.720497 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 29972
I20260620 18:47:32.720803 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:47:32.721287 15771 external_mini_cluster.cc:1428] Setting key e06e63c4c2a77a7a1ea0dea8a175dec4
Jun 20 18:47:32 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981252, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:32.732802 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.713826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:50651 (local address 127.15.102.254:34189)
0620 18:47:32.714346 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:32.714352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:32.715395 (+ 1043us) server_negotiation.cc:407] Connection header received
0620 18:47:32.717379 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:32.717385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:32.717483 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:32.717612 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:32.719360 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.720164 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.722210 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.722689 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.722838 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.726705 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:32.726711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:32.726714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:32.726751 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:32.729351 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:32.730214 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:32.730224 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:32.730227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:32.730369 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:32.730845 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:32.730856 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:32.730859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:32.731143 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:32.731354 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:32.732126 (+ 772us) server_negotiation.cc:299] Negotiation successful
0620 18:47:32.732396 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":256,"threads_started":1}
I20260620 18:47:32.733862 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.714385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:50651)
0620 18:47:32.715151 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:32.715198 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:32.717052 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:32.717899 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:32.717908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:32.718480 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:32.719105 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.719126 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.720390 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.720394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.722034 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.722055 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.722970 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.722977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.723225 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.724024 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:32.724052 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:32.726462 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:32.729542 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:32.729553 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:32.729580 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:32.730061 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:32.730494 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:32.730500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:32.730504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:32.730710 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:32.731309 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:32.731319 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:32.731721 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 18:47:32.732019 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 18:47:32.732625 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":286,"threads_started":1}
I20260620 18:47:32.735710 30100 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34189
I20260620 18:47:32.736114 30100 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:32.737192 30100 heartbeater.cc:511] Master 127.15.102.254:34189 requested a full tablet report, sending...
I20260620 18:47:32.738714 29643 ts_manager.cc:194] Registered new tserver with Master: 86b120ef400f4152a059a9d5189e7a1c (127.15.102.195:44537)
I20260620 18:47:32.739720 29643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:50651
I20260620 18:47:32.750366 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:47:32.769155 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.755381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.15.102.254:34189)
0620 18:47:32.755623 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:32.755633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:32.756028 (+ 395us) server_negotiation.cc:407] Connection header received
0620 18:47:32.756250 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:32.756256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:32.756373 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:32.756489 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:32.757970 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.758729 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.760210 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.760640 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.760788 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.763480 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:32.763485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:32.763488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:32.763528 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:32.766145 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:32.766946 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:32.766951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:32.766955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:32.767098 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:32.767544 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:32.767552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:32.767559 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:32.767916 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:32.768183 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:32.768568 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 18:47:32.768816 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:47:32.773751 29643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:47:32.781164 29643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:47:32.817930 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.795460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:44163 (local address 127.0.0.1:56430)
0620 18:47:32.807567 (+ 12107us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:32.807648 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:32.807963 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:32.809746 (+ 1783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:32.809760 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:32.809812 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:32.810778 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.810799 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.812588 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.812593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.814992 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.815007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.816863 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.816869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.817085 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.817151 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 18:47:32.817297 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 18:47:32.817520 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11636,"thread_start_us":12729,"threads_started":1}
I20260620 18:47:32.817956 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.795912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.15.102.194:44163)
0620 18:47:32.808897 (+ 12985us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:32.808908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:32.808943 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:47:32.809055 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:32.809064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:32.809416 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:32.809661 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:32.811020 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.812434 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.815219 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.816645 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.816828 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.817024 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:32.817422 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 18:47:32.817611 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12776,"thread_start_us":13933,"threads_started":1}
I20260620 18:47:32.819597 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.797004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.15.102.195:44537)
0620 18:47:32.809111 (+ 12107us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:32.809118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:32.810856 (+ 1738us) server_negotiation.cc:407] Connection header received
0620 18:47:32.810964 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:32.810972 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:32.811192 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:32.811435 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:32.812414 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.813515 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.816340 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.817836 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.818049 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.818247 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:32.819047 (+ 800us) server_negotiation.cc:299] Negotiation successful
0620 18:47:32.819219 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11928,"thread_start_us":11753,"threads_started":1}
I20260620 18:47:32.819591 30115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.795813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:44537 (local address 127.0.0.1:49070)
0620 18:47:32.810545 (+ 14732us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:32.810565 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:32.810698 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:32.811770 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:32.811775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:32.811800 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:32.812168 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.812180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.813673 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.813679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.816150 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.816166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.818485 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.818493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.818805 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.818874 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 18:47:32.818971 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 18:47:32.819113 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14593,"mutex_wait_us":135,"thread_start_us":14371,"threads_started":1}
I20260620 18:47:32.820827 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.795484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:41367 (local address 127.0.0.1:44304)
0620 18:47:32.808233 (+ 12749us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:32.808253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:32.808372 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:32.810297 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:32.810303 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:32.810336 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:32.810818 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.810829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.814059 (+ 3230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.814065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.816442 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.816452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.818484 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.818493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.818808 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.820168 (+ 1360us) client_negotiation.cc:769] Sending connection context
0620 18:47:32.820293 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 18:47:32.820438 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12603,"mutex_wait_us":1412,"thread_start_us":11399,"threads_started":1}
I20260620 18:47:32.820880 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.796483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.15.102.193:41367)
0620 18:47:32.809638 (+ 13155us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:32.809647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:32.809674 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:47:32.809790 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:32.809798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:32.810042 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:32.811018 (+ 976us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:32.812290 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.813908 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.816614 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.818134 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.818347 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.818527 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:32.820349 (+ 1822us) server_negotiation.cc:299] Negotiation successful
0620 18:47:32.820545 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12959,"thread_start_us":12783,"threads_started":1}
I20260620 18:47:32.822739 29901 tablet_service.cc:1467] Processing CreateTablet for tablet 98932b9ecbb044e0a09a75302f10d8eb (DEFAULT_TABLE table=test-table [id=161be0f95ee143c9b84d2058abd96dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:32.824091 29901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98932b9ecbb044e0a09a75302f10d8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:32.824903 30035 tablet_service.cc:1467] Processing CreateTablet for tablet 98932b9ecbb044e0a09a75302f10d8eb (DEFAULT_TABLE table=test-table [id=161be0f95ee143c9b84d2058abd96dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:32.824908 29767 tablet_service.cc:1467] Processing CreateTablet for tablet 98932b9ecbb044e0a09a75302f10d8eb (DEFAULT_TABLE table=test-table [id=161be0f95ee143c9b84d2058abd96dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:32.826453 30035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98932b9ecbb044e0a09a75302f10d8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:32.826642 29767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98932b9ecbb044e0a09a75302f10d8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:32.836685 30119 tablet_bootstrap.cc:492] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Bootstrap starting.
I20260620 18:47:32.841522 30119 tablet_bootstrap.cc:654] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:32.842653 30119 log.cc:824] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:32.843762 30121 tablet_bootstrap.cc:492] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: Bootstrap starting.
I20260620 18:47:32.844771 30120 tablet_bootstrap.cc:492] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Bootstrap starting.
I20260620 18:47:32.848450 30121 tablet_bootstrap.cc:654] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:32.849586 30120 tablet_bootstrap.cc:654] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:32.849673 30121 log.cc:824] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:32.851162 30120 log.cc:824] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:32.853282 30119 tablet_bootstrap.cc:492] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: No bootstrap required, opened a new log
I20260620 18:47:32.853597 30119 ts_tablet_manager.cc:1397] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20260620 18:47:32.853597 30121 tablet_bootstrap.cc:492] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: No bootstrap required, opened a new log
I20260620 18:47:32.853940 30121 ts_tablet_manager.cc:1397] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 18:47:32.858577 30120 tablet_bootstrap.cc:492] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: No bootstrap required, opened a new log
I20260620 18:47:32.858922 30120 ts_tablet_manager.cc:1397] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 18:47:32.862535 30119 raft_consensus.cc:348] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.862983 30119 raft_consensus.cc:374] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:32.863085 30119 raft_consensus.cc:729] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a2bc11bbad4c33930efb5a940c3506, State: Initialized, Role: FOLLOWER
I20260620 18:47:32.863838 30119 consensus_queue.cc:260] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.865108 29966 heartbeater.cc:503] Master 127.15.102.254:34189 was elected leader, sending a full tablet report...
I20260620 18:47:32.866019 30119 ts_tablet_manager.cc:1428] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260620 18:47:32.865723 30121 raft_consensus.cc:348] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.866192 30121 raft_consensus.cc:374] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:32.866304 30121 raft_consensus.cc:729] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88dd7880435b43c39f624069b8025012, State: Initialized, Role: FOLLOWER
I20260620 18:47:32.866971 30121 consensus_queue.cc:260] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.871884 30120 raft_consensus.cc:348] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.873077 30121 ts_tablet_manager.cc:1428] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: Time spent starting tablet: real 0.019s user 0.015s sys 0.001s
I20260620 18:47:32.873163 30120 raft_consensus.cc:374] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:32.873286 30120 raft_consensus.cc:729] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b120ef400f4152a059a9d5189e7a1c, State: Initialized, Role: FOLLOWER
I20260620 18:47:32.873955 30120 consensus_queue.cc:260] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.875134 30100 heartbeater.cc:503] Master 127.15.102.254:34189 was elected leader, sending a full tablet report...
I20260620 18:47:32.876052 30120 ts_tablet_manager.cc:1428] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260620 18:47:32.919950 29967 tablet.cc:2369] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:47:32.962800 30101 tablet.cc:2369] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:32.965758 30126 raft_consensus.cc:482] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:47:32.966095 30126 raft_consensus.cc:504] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.968115 30126 leader_election.cc:288] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44a2bc11bbad4c33930efb5a940c3506 (127.15.102.194:44163), 86b120ef400f4152a059a9d5189e7a1c (127.15.102.195:44537)
I20260620 18:47:32.978155 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.968603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:60641 (local address 127.15.102.194:44163)
0620 18:47:32.968823 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:32.968830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:32.969623 (+ 793us) server_negotiation.cc:407] Connection header received
0620 18:47:32.969844 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:32.969853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:32.969945 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:32.970051 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:32.970992 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.972188 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.975503 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.976810 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.977004 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.977086 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:32.977715 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 18:47:32.977875 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:47:32.978155 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.968694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:59663 (local address 127.15.102.195:44537)
0620 18:47:32.968901 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:32.968910 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:32.969623 (+ 713us) server_negotiation.cc:407] Connection header received
0620 18:47:32.969985 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:32.969994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:32.970077 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:32.970167 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:32.970971 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.972190 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.975502 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.976810 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.977004 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.977086 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:32.977715 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 18:47:32.977879 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 18:47:32.978474 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.968515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:44163 (local address 127.15.102.193:60641)
0620 18:47:32.969499 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:32.969513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:32.969663 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:32.970083 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:32.970087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:32.970116 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:32.970830 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.970844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.972353 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.972359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.975219 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.975274 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.977154 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.977162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.977430 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.977505 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 18:47:32.977619 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:47:32.977807 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":362,"thread_start_us":508,"threads_started":1}
I20260620 18:47:32.978648 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:32.968437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:44537 (local address 127.15.102.193:59663)
0620 18:47:32.969465 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:32.969495 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:32.969703 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:32.970203 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:32.970207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:32.970277 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:32.970757 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.970779 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.972353 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.972359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.975219 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:32.975274 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:32.977166 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:32.977171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:32.977431 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:32.977499 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 18:47:32.977619 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 18:47:32.977808 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":201,"spinlock_wait_cycles":56320,"thread_start_us":657,"threads_started":1}
I20260620 18:47:32.979764 29921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98932b9ecbb044e0a09a75302f10d8eb" candidate_uuid: "88dd7880435b43c39f624069b8025012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a2bc11bbad4c33930efb5a940c3506" is_pre_election: true
I20260620 18:47:32.979849 30055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98932b9ecbb044e0a09a75302f10d8eb" candidate_uuid: "88dd7880435b43c39f624069b8025012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b120ef400f4152a059a9d5189e7a1c" is_pre_election: true
I20260620 18:47:32.980420 30055 raft_consensus.cc:2454] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88dd7880435b43c39f624069b8025012 in term 0.
I20260620 18:47:32.980422 29921 raft_consensus.cc:2454] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88dd7880435b43c39f624069b8025012 in term 0.
I20260620 18:47:32.981276 29723 leader_election.cc:302] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 44a2bc11bbad4c33930efb5a940c3506, 86b120ef400f4152a059a9d5189e7a1c, 88dd7880435b43c39f624069b8025012; no voters:
I20260620 18:47:32.981777 30126 raft_consensus.cc:2781] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:47:32.981936 30126 raft_consensus.cc:482] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:47:32.982064 30126 raft_consensus.cc:3037] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:32.984537 30126 raft_consensus.cc:504] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.985325 30126 leader_election.cc:288] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [CANDIDATE]: Term 1 election: Requested vote from peers 44a2bc11bbad4c33930efb5a940c3506 (127.15.102.194:44163), 86b120ef400f4152a059a9d5189e7a1c (127.15.102.195:44537)
I20260620 18:47:32.985930 29921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98932b9ecbb044e0a09a75302f10d8eb" candidate_uuid: "88dd7880435b43c39f624069b8025012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a2bc11bbad4c33930efb5a940c3506"
I20260620 18:47:32.986001 30055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98932b9ecbb044e0a09a75302f10d8eb" candidate_uuid: "88dd7880435b43c39f624069b8025012" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b120ef400f4152a059a9d5189e7a1c"
I20260620 18:47:32.986212 29921 raft_consensus.cc:3037] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:32.986245 30055 raft_consensus.cc:3037] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:32.988698 30055 raft_consensus.cc:2454] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88dd7880435b43c39f624069b8025012 in term 1.
I20260620 18:47:32.988718 29921 raft_consensus.cc:2454] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88dd7880435b43c39f624069b8025012 in term 1.
I20260620 18:47:32.989241 29723 leader_election.cc:302] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b120ef400f4152a059a9d5189e7a1c, 88dd7880435b43c39f624069b8025012; no voters:
I20260620 18:47:32.989611 30126 raft_consensus.cc:2781] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:32.990113 30126 raft_consensus.cc:686] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 1 LEADER]: Becoming Leader. State: Replica: 88dd7880435b43c39f624069b8025012, State: Running, Role: LEADER
I20260620 18:47:32.990716 30126 consensus_queue.cc:237] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:32.996078 29641 catalog_manager.cc:5495] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88dd7880435b43c39f624069b8025012 (127.15.102.193). New cstate: current_term: 1 leader_uuid: "88dd7880435b43c39f624069b8025012" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } health_report { overall_health: UNKNOWN } } }
I20260620 18:47:33.014683 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.008609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.15.102.193:41367)
0620 18:47:33.008836 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:33.008844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:33.008870 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:47:33.009176 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:33.009180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:33.009261 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:33.009435 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:33.009842 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.010562 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.012077 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.012709 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.012868 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.013463 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:33.014008 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:33.014187 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:33.014305 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:47:33.014401 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:47:33.024044 30055 raft_consensus.cc:1261] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 1 FOLLOWER]: Refusing update from remote peer 88dd7880435b43c39f624069b8025012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:33.024045 29921 raft_consensus.cc:1261] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 1 FOLLOWER]: Refusing update from remote peer 88dd7880435b43c39f624069b8025012: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:33.025242 30130 consensus_queue.cc:1035] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:33.025657 30126 consensus_queue.cc:1035] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:33.041286 30135 mvcc.cc:205] Tried to move back new op lower bound from 7298995212374798336 to 7298995212254179328. Current Snapshot: MvccSnapshot[applied={T|T < 7298995212374798336}]
I20260620 18:47:33.058668 29642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:47:33.059042 29642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:47:33.063977 29642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Sending DeleteTablet for 3 replicas of tablet 98932b9ecbb044e0a09a75302f10d8eb
I20260620 18:47:33.065399 29901 tablet_service.cc:1514] Processing DeleteTablet for tablet 98932b9ecbb044e0a09a75302f10d8eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:33 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:56430
I20260620 18:47:33.065676 30035 tablet_service.cc:1514] Processing DeleteTablet for tablet 98932b9ecbb044e0a09a75302f10d8eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:33 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:49070
I20260620 18:47:33.066164 29767 tablet_service.cc:1514] Processing DeleteTablet for tablet 98932b9ecbb044e0a09a75302f10d8eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:33 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:44304
I20260620 18:47:33.066864 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 29704
I20260620 18:47:33.077943 30145 tablet_replica.cc:330] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: stopping tablet replica
I20260620 18:47:33.078148 30143 tablet_replica.cc:330] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: stopping tablet replica
I20260620 18:47:33.078683 30143 raft_consensus.cc:2229] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:47:33.078703 30145 raft_consensus.cc:2229] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:47:33.079368 30143 raft_consensus.cc:2258] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:33.079623 30145 raft_consensus.cc:2258] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:33.081250 30144 tablet_replica.cc:330] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: stopping tablet replica
I20260620 18:47:33.081780 30144 raft_consensus.cc:2229] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:47:33.082401 30144 raft_consensus.cc:2258] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:33.083031 30145 ts_tablet_manager.cc:1905] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:33.084564 30143 ts_tablet_manager.cc:1905] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:33.087852 30144 ts_tablet_manager.cc:1905] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:33.092198 30143 ts_tablet_manager.cc:1918] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:33.092464 30143 log.cc:1196] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/98932b9ecbb044e0a09a75302f10d8eb
I20260620 18:47:33.093063 30143 ts_tablet_manager.cc:1939] T 98932b9ecbb044e0a09a75302f10d8eb P 44a2bc11bbad4c33930efb5a940c3506: Deleting consensus metadata
Jun 20 18:47:33 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:47:33.096051 29627 catalog_manager.cc:4841] TS 44a2bc11bbad4c33930efb5a940c3506 (127.15.102.194:44163): tablet 98932b9ecbb044e0a09a75302f10d8eb (table test-table [id=161be0f95ee143c9b84d2058abd96dcf]) successfully deleted
I20260620 18:47:33.096880 30145 ts_tablet_manager.cc:1918] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:33.097143 30145 log.cc:1196] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/98932b9ecbb044e0a09a75302f10d8eb
I20260620 18:47:33.097783 30145 ts_tablet_manager.cc:1939] T 98932b9ecbb044e0a09a75302f10d8eb P 88dd7880435b43c39f624069b8025012: Deleting consensus metadata
I20260620 18:47:33.099869 29629 catalog_manager.cc:4841] TS 88dd7880435b43c39f624069b8025012 (127.15.102.193:41367): tablet 98932b9ecbb044e0a09a75302f10d8eb (table test-table [id=161be0f95ee143c9b84d2058abd96dcf]) successfully deleted
I20260620 18:47:33.100193 30144 ts_tablet_manager.cc:1918] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:33.100389 30144 log.cc:1196] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/98932b9ecbb044e0a09a75302f10d8eb
I20260620 18:47:33.100870 30144 ts_tablet_manager.cc:1939] T 98932b9ecbb044e0a09a75302f10d8eb P 86b120ef400f4152a059a9d5189e7a1c: Deleting consensus metadata
I20260620 18:47:33.102842 29630 catalog_manager.cc:4841] TS 86b120ef400f4152a059a9d5189e7a1c (127.15.102.195:44537): tablet 98932b9ecbb044e0a09a75302f10d8eb (table test-table [id=161be0f95ee143c9b84d2058abd96dcf]) successfully deleted
I20260620 18:47:33.103837 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.068186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.15.102.193:41367)
0620 18:47:33.068398 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:33.068405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:33.080804 (+ 12399us) server_negotiation.cc:407] Connection header received
0620 18:47:33.080910 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:33.080916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:33.081016 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:33.081140 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:33.083575 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.084826 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.087088 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.087853 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.087977 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.094631 (+ 6654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:33.094651 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:33.094672 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:33.094749 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:33.099975 (+ 5226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.101148 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.101160 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.101163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.101368 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.101880 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.101886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.101889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.102397 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:33.102700 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:33.103156 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 18:47:33.103493 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:47:33.104629 29827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:33.206634 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 29704
I20260620 18:47:33.226643 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 29838
Jun 20 18:47:33 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:47:33.244889 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.227579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.15.102.194:44163)
0620 18:47:33.227843 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:33.227850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:33.227874 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:47:33.228023 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:33.228028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:33.228112 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:33.228240 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:33.229771 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.230619 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.232186 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.232635 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.232813 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.237756 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:33.237778 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:33.237795 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:33.237844 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:33.241219 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.242168 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.242178 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.242181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.242367 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.242869 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.242874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.242876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.243448 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:33.243821 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:33.244188 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 18:47:33.244460 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 18:47:33.245709 29961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:33.337703 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 29838
I20260620 18:47:33.355863 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 29972
Jun 20 18:47:33 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:47:33.373504 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:33.356673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.15.102.195:44537)
0620 18:47:33.356852 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:33.356857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:33.356893 (+ 36us) server_negotiation.cc:407] Connection header received
0620 18:47:33.357076 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:33.357081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:33.357150 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:33.357284 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:33.358708 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.359532 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.361067 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:33.361472 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:33.361571 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:33.366834 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:33.366854 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:33.366871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:33.366914 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:33.370353 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.371363 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.371372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.371375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.371516 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:33.371907 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:33.371910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:33.371912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:33.372274 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:33.372527 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:33.372988 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 18:47:33.373191 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:47:33.374310 30095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:33.469476 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 29972
I20260620 18:47:33.487311 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 29611
I20260620 18:47:33.488684 29673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:33.603603 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 29611
I20260620 18:47:33.625195 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:34189
--webserver_interface=127.15.102.254
--webserver_port=34503
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:33.980788 30151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:33.981133 30151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:33.981304 30151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:33.989073 30151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:33.989223 30151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:33.989344 30151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:33.989427 30151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:33.989494 30151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:34.000180 30151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=34503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:34.001803 30151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:34.003461 30151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:34.029698 30158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:34.029613 30157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:34.029647 30160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:34.031077 30151 server_base.cc:1034] running on GCE node
I20260620 18:47:34.032182 30151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:34.034176 30151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:34.035477 30151 hybrid_clock.cc:648] HybridClock initialized: now 1781981254035428 us; error 82 us; skew 500 ppm
I20260620 18:47:34.035955 30151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:34 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981254, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:34.040120 30151 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:47:34.042294 30151 webserver.cc:460] Webserver started at http://127.15.102.254:34503/ using document root <none> and password file <none>
I20260620 18:47:34.043347 30151 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:34.043558 30151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:34.050853 30151 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:47:34.054545 30167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:34.056411 30151 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 18:47:34.056672 30151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "394f633ec05c4078a6a98e8273b12c70"
format_stamp: "Formatted at 2026-06-20 18:47:30 on dist-test-slave-zfp8"
server_key: "81e4d7cb58f21d8fa9bbf3767bc2b39c"
server_key_iv: "07e4583bb4a247dc42f54a856ff27fd5"
server_key_version: "encryptionkey@0"
I20260620 18:47:34.057920 30151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:34.072767 30151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:34.079106 30151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:34.079612 30151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:34.102605 30151 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:34189
I20260620 18:47:34.102612 30218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:34189 every 8 connection(s)
I20260620 18:47:34.104480 30151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:34.110782 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 30151
I20260620 18:47:34.111910 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:41367
--local_ip_for_outbound_sockets=127.15.102.193
--tserver_master_addrs=127.15.102.254:34189
--webserver_port=35453
--webserver_interface=127.15.102.193
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:34.113726 30219 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:47:34.120042 30219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Bootstrap starting.
I20260620 18:47:34.142748 30219 log.cc:824] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:34.157763 30219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:47:34.158396 30219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Bootstrap complete.
I20260620 18:47:34.168869 30219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:34.169929 30219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394f633ec05c4078a6a98e8273b12c70, State: Initialized, Role: FOLLOWER
I20260620 18:47:34.170676 30219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:34.170979 30219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:34.171134 30219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:34.171386 30219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:47:34.175990 30219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:34.176688 30219 leader_election.cc:302] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 394f633ec05c4078a6a98e8273b12c70; no voters:
I20260620 18:47:34.177424 30219 leader_election.cc:288] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:47:34.177574 30223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:47:34.179088 30223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 2 LEADER]: Becoming Leader. State: Replica: 394f633ec05c4078a6a98e8273b12c70, State: Running, Role: LEADER
I20260620 18:47:34.179821 30223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:34.180269 30219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:34.182859 30224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "394f633ec05c4078a6a98e8273b12c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } } }
I20260620 18:47:34.183458 30224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:34.183492 30225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394f633ec05c4078a6a98e8273b12c70. Latest consensus state: current_term: 2 leader_uuid: "394f633ec05c4078a6a98e8273b12c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } } }
I20260620 18:47:34.187573 30225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:34.188634 30229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:34.199383 30229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:34.201128 30229 catalog_manager.cc:1259] Loaded cluster ID: 557950285866474297c662a6d5afbd56
I20260620 18:47:34.201340 30229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:34.205020 30229 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 18:47:34.215598 30240 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:47:34.223263 30229 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 18:47:34.233203 30229 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:34.235154 30229 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Loaded TSK: 0
I20260620 18:47:34.236959 30229 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:47:34.499837 30221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:34.500164 30221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:34.500334 30221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:34.508643 30221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:34.508771 30221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:34.508953 30221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:47:34.519605 30221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:41367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=35453
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:34.521083 30221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:34.522567 30221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:34.537201 30247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:34.537242 30249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:34.537396 30246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:34.538163 30221 server_base.cc:1034] running on GCE node
I20260620 18:47:34.539047 30221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:34.540436 30221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:34.541733 30221 hybrid_clock.cc:648] HybridClock initialized: now 1781981254541638 us; error 67 us; skew 500 ppm
I20260620 18:47:34.542147 30221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:34 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981254, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:34.546363 30221 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:47:34.548290 30221 webserver.cc:460] Webserver started at http://127.15.102.193:35453/ using document root <none> and password file <none>
I20260620 18:47:34.549124 30221 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:34.549297 30221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:34.556140 30221 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:47:34.559832 30256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:34.561733 30221 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:47:34.561978 30221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "88dd7880435b43c39f624069b8025012"
format_stamp: "Formatted at 2026-06-20 18:47:31 on dist-test-slave-zfp8"
server_key: "a85d9409411090332545321e7169c8a8"
server_key_iv: "7e530f3589941b6a51842cb0f1fdf149"
server_key_version: "encryptionkey@0"
I20260620 18:47:34.563359 30221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:34.594082 30221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:34.599467 30221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:34.599800 30221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:34.601016 30221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:34.602713 30221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:34.602878 30221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:34.603041 30221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:34.603140 30221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:34.639843 30221 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:41367
I20260620 18:47:34.639948 30368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:41367 every 8 connection(s)
I20260620 18:47:34.641484 30221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:47:34.651739 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 30221
I20260620 18:47:34.652523 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:44163
--local_ip_for_outbound_sockets=127.15.102.194
--tserver_master_addrs=127.15.102.254:34189
--webserver_port=33611
--webserver_interface=127.15.102.194
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:47:34 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981254, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:34.669682 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.644901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:46827 (local address 127.15.102.254:34189)
0620 18:47:34.645511 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:34.645529 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:34.646446 (+ 917us) server_negotiation.cc:407] Connection header received
0620 18:47:34.648803 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:34.648819 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:34.649225 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:34.649596 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:34.651375 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.653009 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.654895 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.655718 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.655973 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.660236 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:34.660246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:34.660270 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:34.660316 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:34.664024 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.665273 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.665282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.665288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.665498 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:34.666129 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:34.666133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:34.666135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:34.666675 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:34.666836 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:34.668112 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0620 18:47:34.668515 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":221,"threads_started":1}
I20260620 18:47:34.670020 30372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:34.645345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:46827)
0620 18:47:34.646197 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:34.646241 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:34.648162 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:34.649755 (+ 1593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:34.649763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:34.650356 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:34.650992 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.651016 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.653278 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.653283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.654665 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:34.654696 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:34.656021 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:34.656029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:34.656335 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:34.657128 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:34.657153 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:34.659843 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:34.664254 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.664269 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.664291 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.665070 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.665662 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:34.665671 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:34.665677 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:34.666006 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:34.666856 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:34.666869 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:34.667459 (+ 590us) client_negotiation.cc:769] Sending connection context
0620 18:47:34.667862 (+ 403us) client_negotiation.cc:240] Negotiation successful
0620 18:47:34.668552 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":337,"threads_started":1}
I20260620 18:47:34.673800 30369 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34189
I20260620 18:47:34.674178 30369 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:34.675067 30369 heartbeater.cc:511] Master 127.15.102.254:34189 requested a full tablet report, sending...
I20260620 18:47:34.678124 30184 ts_manager.cc:194] Registered new tserver with Master: 88dd7880435b43c39f624069b8025012 (127.15.102.193:41367)
I20260620 18:47:34.680447 30184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:46827
W20260620 18:47:35.008322 30373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:35.008745 30373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:35.008929 30373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:35.016528 30373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:35.016662 30373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:35.016852 30373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:47:35.027109 30373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:44163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=33611
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:35.028654 30373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:35.030174 30373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:35.045163 30379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:35.045181 30381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:35.045106 30378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:35.046236 30373 server_base.cc:1034] running on GCE node
I20260620 18:47:35.047006 30373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:35.048195 30373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:35.049443 30373 hybrid_clock.cc:648] HybridClock initialized: now 1781981255049373 us; error 66 us; skew 500 ppm
I20260620 18:47:35.049823 30373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:35 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981255, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:35.053431 30373 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:47:35.055094 30373 webserver.cc:460] Webserver started at http://127.15.102.194:33611/ using document root <none> and password file <none>
I20260620 18:47:35.055975 30373 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:35.056154 30373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:35.062194 30373 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 18:47:35.065323 30388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:35.067003 30373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:47:35.067207 30373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "44a2bc11bbad4c33930efb5a940c3506"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "1ba05c92acdaad88ea85d94f51a69477"
server_key_iv: "c63e6e684f637c955808bdfe9a802f95"
server_key_version: "encryptionkey@0"
I20260620 18:47:35.068483 30373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:35.108603 30373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:35.114408 30373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:35.114691 30373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:35.115864 30373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:35.117548 30373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:35.117650 30373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:35.117754 30373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:35.117838 30373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:35.153539 30373 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:44163
I20260620 18:47:35.153635 30500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:44163 every 8 connection(s)
I20260620 18:47:35.155040 30373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:47:35.160701 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 30373
I20260620 18:47:35.161499 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:44537
--local_ip_for_outbound_sockets=127.15.102.195
--tserver_master_addrs=127.15.102.254:34189
--webserver_port=36973
--webserver_interface=127.15.102.195
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:47:35 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981255, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:35.177438 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.157703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:49065 (local address 127.15.102.254:34189)
0620 18:47:35.157904 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.157909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.159049 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 18:47:35.161043 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.161049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.161173 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.161298 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:35.163073 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.164317 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.165947 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.166740 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.166911 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.171039 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:35.171044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:35.171048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:35.171085 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:35.173853 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:35.174872 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:35.174877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:35.174879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:35.175025 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:35.175589 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:35.175593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:35.175595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:35.175885 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:35.176003 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.176983 (+ 980us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.177177 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:47:35.179085 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.158133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:49065)
0620 18:47:35.158863 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:35.158902 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:35.160637 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:35.161520 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:35.161530 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:35.162258 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:35.162879 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.162898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.164486 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.164490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.165778 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.165791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.167017 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.167023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.167297 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.168085 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:35.168111 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:35.170762 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:35.174009 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:35.174021 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:35.174043 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:35.174711 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:35.175148 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:35.175155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:35.175162 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:35.175451 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:35.176022 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:35.176033 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:35.176545 (+ 512us) client_negotiation.cc:769] Sending connection context
0620 18:47:35.176866 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 18:47:35.177487 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":266,"threads_started":1}
I20260620 18:47:35.180855 30501 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34189
I20260620 18:47:35.181210 30501 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:35.182106 30501 heartbeater.cc:511] Master 127.15.102.254:34189 requested a full tablet report, sending...
I20260620 18:47:35.183792 30184 ts_manager.cc:194] Registered new tserver with Master: 44a2bc11bbad4c33930efb5a940c3506 (127.15.102.194:44163)
I20260620 18:47:35.184628 30184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:49065
W20260620 18:47:35.503099 30504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:35.503518 30504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:35.503695 30504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:35.511541 30504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:35.511659 30504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:35.511838 30504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:47:35.521910 30504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:44537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=36973
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:35.523366 30504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:35.524741 30504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:35.538736 30509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:35.538708 30510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:35.538708 30512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:35.539961 30504 server_base.cc:1034] running on GCE node
I20260620 18:47:35.540616 30504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:35.541612 30504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:35.542816 30504 hybrid_clock.cc:648] HybridClock initialized: now 1781981255542772 us; error 41 us; skew 500 ppm
I20260620 18:47:35.543152 30504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:35 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981255, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:35.546476 30504 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:47:35.548280 30504 webserver.cc:460] Webserver started at http://127.15.102.195:36973/ using document root <none> and password file <none>
I20260620 18:47:35.549029 30504 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:35.549198 30504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:35.555348 30504 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 18:47:35.558477 30519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:35.560189 30504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:47:35.560416 30504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "86b120ef400f4152a059a9d5189e7a1c"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "ca4449eee88d5050348af4828b5ff4ee"
server_key_iv: "919363098dadf269a4f4299d7613c304"
server_key_version: "encryptionkey@0"
I20260620 18:47:35.561565 30504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:35.576530 30504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:35.581539 30504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:35.581892 30504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:35.582996 30504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:35.584681 30504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:35.584805 30504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:35.584946 30504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:35.585044 30504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:35.620350 30504 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:44537
I20260620 18:47:35.620465 30631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:44537 every 8 connection(s)
I20260620 18:47:35.621937 30504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:47:35.629987 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 30504
Jun 20 18:47:35 dist-test-slave-zfp8 krb5kdc[29598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981255, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:35.645787 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.624965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:56545 (local address 127.15.102.254:34189)
0620 18:47:35.625202 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.625207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.626539 (+ 1332us) server_negotiation.cc:407] Connection header received
0620 18:47:35.628572 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.628576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.628667 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.628769 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:35.630511 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.631625 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.633605 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.634161 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.634268 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.638377 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:35.638384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:35.638392 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:35.638433 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:35.642001 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:35.643036 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:35.643044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:35.643048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:35.643205 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:35.643859 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:35.643866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:35.643875 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:35.644203 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:35.644388 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.645197 (+ 809us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.645439 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:47:35.647215 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.625468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:56545)
0620 18:47:35.626337 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:35.626386 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:35.628259 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:35.629024 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:35.629034 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:35.629600 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:35.630299 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.630320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.631845 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.631849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.633432 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.633450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.634411 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.634420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.634734 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.635606 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:35.635631 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:35.638125 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:35.642212 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:35.642222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:35.642247 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:35.642880 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:35.643376 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:35.643381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:35.643385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:35.643693 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:35.644370 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:35.644379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:35.644779 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 18:47:35.645084 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 18:47:35.645694 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":372,"threads_started":1}
I20260620 18:47:35.648977 30632 heartbeater.cc:347] Connected to a master server at 127.15.102.254:34189
I20260620 18:47:35.649335 30632 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:35.649339 30635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.631009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.15.102.254:34189)
0620 18:47:35.631649 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.631655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.631949 (+ 294us) server_negotiation.cc:407] Connection header received
0620 18:47:35.632082 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.632087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.632168 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.632292 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:35.633811 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.634866 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.636512 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.637111 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.637212 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.641318 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:35.641327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:35.641331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:35.641392 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:35.645359 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:35.646965 (+ 1606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:35.646969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:35.646971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:35.647095 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:35.647597 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:35.647604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:35.647607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:35.648048 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:35.648326 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.648691 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.648949 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":314,"threads_started":1}
I20260620 18:47:35.650347 30632 heartbeater.cc:511] Master 127.15.102.254:34189 requested a full tablet report, sending...
I20260620 18:47:35.652482 30184 ts_manager.cc:194] Registered new tserver with Master: 86b120ef400f4152a059a9d5189e7a1c (127.15.102.195:44537)
I20260620 18:47:35.653355 30184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:56545
I20260620 18:47:35.664929 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:47:35.672326 30635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.665945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.15.102.254:34189)
0620 18:47:35.666183 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.666193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.666620 (+ 427us) server_negotiation.cc:407] Connection header received
0620 18:47:35.666829 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.666834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.666996 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.667134 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:35.667680 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.668488 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.669929 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.670387 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.670486 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.671039 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:35.671697 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:35.671830 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.671930 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.672011 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:47:35.673507 30184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:47:35.681350 30184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:47:35.684314 30369 heartbeater.cc:503] Master 127.15.102.254:34189 was elected leader, sending a full tablet report...
I20260620 18:47:35.720774 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.695021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.15.102.194:44163)
0620 18:47:35.709005 (+ 13984us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.709020 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.709063 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:47:35.709201 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.709214 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.709583 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.710509 (+ 926us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:35.711715 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.713407 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.717526 (+ 4119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.719034 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.719189 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.719412 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.720211 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.720393 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13728,"thread_start_us":14810,"threads_started":1}
I20260620 18:47:35.720862 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.694587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:41367 (local address 127.0.0.1:44332)
0620 18:47:35.703903 (+ 9316us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:35.703947 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:35.704201 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:35.711428 (+ 7227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:35.711440 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:35.711466 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:35.711886 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.711896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.713971 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.713977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.717203 (+ 3226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.717216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.719833 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.719843 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.719998 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.720025 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 18:47:35.720147 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:47:35.720387 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9045,"mutex_wait_us":181,"thread_start_us":21408,"threads_started":1}
I20260620 18:47:35.721138 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.694805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.15.102.193:41367)
0620 18:47:35.710298 (+ 15493us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.710310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.710345 (+ 35us) server_negotiation.cc:407] Connection header received
0620 18:47:35.710456 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.710464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.710728 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.711152 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:35.712105 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.713615 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.717435 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.719027 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.719221 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.719809 (+ 588us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.720627 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.720822 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14860,"thread_start_us":15203,"threads_started":1}
I20260620 18:47:35.721355 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.694788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:44163 (local address 127.0.0.1:56444)
0620 18:47:35.706538 (+ 11750us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:35.706565 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:35.706742 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:35.710815 (+ 4073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:35.710823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:35.710856 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:35.711473 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.711487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.713822 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.713828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.717309 (+ 3481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.717329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.719565 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.719573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.719898 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.719974 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 18:47:35.720111 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 18:47:35.720411 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11556,"mutex_wait_us":465,"thread_start_us":11345,"threads_started":1}
I20260620 18:47:35.724663 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.696729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:44537 (local address 127.0.0.1:49090)
0620 18:47:35.716359 (+ 19630us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:35.716380 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:35.716509 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:35.717158 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:35.717165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:35.717191 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:35.717608 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.717619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.719544 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.719550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.721955 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.721968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.723603 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.723609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.724012 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.724050 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:47:35.724137 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 18:47:35.724294 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19474,"thread_start_us":6495,"threads_started":1}
I20260620 18:47:35.724679 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.697415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.15.102.195:44537)
0620 18:47:35.715440 (+ 18025us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.715454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.716496 (+ 1042us) server_negotiation.cc:407] Connection header received
0620 18:47:35.716641 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.716649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.716937 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.717157 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:35.717788 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.719383 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.722129 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.723368 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.723899 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.724114 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.724201 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.724371 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16311,"thread_start_us":17499,"threads_started":1}
I20260620 18:47:35.726008 30304 tablet_service.cc:1467] Processing CreateTablet for tablet 4675b4ce9df04c35b3a45256990316f9 (DEFAULT_TABLE table=test-table [id=94bbb5beb52b4175a67977299c18e9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:35.726331 30436 tablet_service.cc:1467] Processing CreateTablet for tablet 4675b4ce9df04c35b3a45256990316f9 (DEFAULT_TABLE table=test-table [id=94bbb5beb52b4175a67977299c18e9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:35.727689 30436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4675b4ce9df04c35b3a45256990316f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:35.727685 30304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4675b4ce9df04c35b3a45256990316f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:35.728400 30567 tablet_service.cc:1467] Processing CreateTablet for tablet 4675b4ce9df04c35b3a45256990316f9 (DEFAULT_TABLE table=test-table [id=94bbb5beb52b4175a67977299c18e9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:35.729521 30567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4675b4ce9df04c35b3a45256990316f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:35.742630 30648 tablet_bootstrap.cc:492] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Bootstrap starting.
I20260620 18:47:35.743723 30649 tablet_bootstrap.cc:492] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Bootstrap starting.
I20260620 18:47:35.745446 30650 tablet_bootstrap.cc:492] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Bootstrap starting.
I20260620 18:47:35.747742 30648 tablet_bootstrap.cc:654] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:35.747745 30649 tablet_bootstrap.cc:654] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:35.748593 30650 tablet_bootstrap.cc:654] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:35.749225 30649 log.cc:824] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:35.749320 30648 log.cc:824] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:35.749897 30650 log.cc:824] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:35.752746 30648 tablet_bootstrap.cc:492] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: No bootstrap required, opened a new log
I20260620 18:47:35.752758 30649 tablet_bootstrap.cc:492] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: No bootstrap required, opened a new log
I20260620 18:47:35.752737 30650 tablet_bootstrap.cc:492] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: No bootstrap required, opened a new log
I20260620 18:47:35.753041 30649 ts_tablet_manager.cc:1397] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 18:47:35.753041 30648 ts_tablet_manager.cc:1397] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 18:47:35.753060 30650 ts_tablet_manager.cc:1397] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 18:47:35.760723 30650 raft_consensus.cc:348] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.761157 30650 raft_consensus.cc:374] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:35.761284 30650 raft_consensus.cc:729] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a2bc11bbad4c33930efb5a940c3506, State: Initialized, Role: FOLLOWER
I20260620 18:47:35.761996 30650 consensus_queue.cc:260] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.763413 30501 heartbeater.cc:503] Master 127.15.102.254:34189 was elected leader, sending a full tablet report...
I20260620 18:47:35.764046 30650 ts_tablet_manager.cc:1428] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 18:47:35.764252 30649 raft_consensus.cc:348] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.764263 30648 raft_consensus.cc:348] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.764729 30648 raft_consensus.cc:374] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:35.764741 30649 raft_consensus.cc:374] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:35.764845 30649 raft_consensus.cc:729] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88dd7880435b43c39f624069b8025012, State: Initialized, Role: FOLLOWER
I20260620 18:47:35.764883 30648 raft_consensus.cc:729] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b120ef400f4152a059a9d5189e7a1c, State: Initialized, Role: FOLLOWER
I20260620 18:47:35.765515 30648 consensus_queue.cc:260] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.765512 30649 consensus_queue.cc:260] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.767172 30632 heartbeater.cc:503] Master 127.15.102.254:34189 was elected leader, sending a full tablet report...
I20260620 18:47:35.767448 30649 ts_tablet_manager.cc:1428] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260620 18:47:35.767853 30648 ts_tablet_manager.cc:1428] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260620 18:47:35.873713 30633 tablet.cc:2369] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:47:35.894120 30370 tablet.cc:2369] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:47:35.907161 30502 tablet.cc:2369] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:35.947711 30655 raft_consensus.cc:482] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:47:35.947971 30655 raft_consensus.cc:504] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.949600 30655 leader_election.cc:288] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88dd7880435b43c39f624069b8025012 (127.15.102.193:41367), 44a2bc11bbad4c33930efb5a940c3506 (127.15.102.194:44163)
I20260620 18:47:35.955572 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.950196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:59299 (local address 127.15.102.194:44163)
0620 18:47:35.950423 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.950429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.950448 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:47:35.950650 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.950661 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.950726 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.950818 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:35.951272 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.952097 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.953873 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.954630 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.954804 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.954896 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.955170 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.955315 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 18:47:35.955575 30634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.950086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:44163 (local address 127.15.102.195:59299)
0620 18:47:35.950334 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:35.950354 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:35.950484 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:35.950855 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:35.950859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:35.950896 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:35.951091 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.951099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.952228 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.952237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.953721 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.953733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.954793 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.954799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.954999 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.955023 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:47:35.955133 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 18:47:35.955290 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":1}
I20260620 18:47:35.956631 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.950570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:41367 (local address 127.15.102.195:59729)
0620 18:47:35.951315 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:35.951329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:35.951464 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:35.951792 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:35.951796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:35.951818 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:35.952120 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.952128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.953186 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.953192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.954620 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:35.954632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.955999 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.956005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:35.956193 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.956217 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:47:35.956292 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 18:47:35.956376 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":21,"thread_start_us":416,"threads_started":1}
I20260620 18:47:35.956732 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:35.950189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:59729 (local address 127.15.102.193:41367)
0620 18:47:35.950424 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:35.950428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:35.951415 (+ 987us) server_negotiation.cc:407] Connection header received
0620 18:47:35.951611 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:35.951615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:35.951674 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:35.951778 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:35.952324 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.953020 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.954786 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:35.955751 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:35.955896 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:35.955964 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:35.956355 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 18:47:35.956487 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:47:35.956660 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4675b4ce9df04c35b3a45256990316f9" candidate_uuid: "86b120ef400f4152a059a9d5189e7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a2bc11bbad4c33930efb5a940c3506" is_pre_election: true
I20260620 18:47:35.957278 30456 raft_consensus.cc:2454] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b120ef400f4152a059a9d5189e7a1c in term 0.
I20260620 18:47:35.957792 30324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4675b4ce9df04c35b3a45256990316f9" candidate_uuid: "86b120ef400f4152a059a9d5189e7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88dd7880435b43c39f624069b8025012" is_pre_election: true
I20260620 18:47:35.958000 30520 leader_election.cc:302] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a2bc11bbad4c33930efb5a940c3506, 86b120ef400f4152a059a9d5189e7a1c; no voters:
I20260620 18:47:35.958343 30324 raft_consensus.cc:2454] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b120ef400f4152a059a9d5189e7a1c in term 0.
I20260620 18:47:35.958432 30655 raft_consensus.cc:2781] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:47:35.958570 30655 raft_consensus.cc:482] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:47:35.958679 30655 raft_consensus.cc:3037] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:35.961964 30655 raft_consensus.cc:504] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.962841 30655 leader_election.cc:288] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [CANDIDATE]: Term 1 election: Requested vote from peers 88dd7880435b43c39f624069b8025012 (127.15.102.193:41367), 44a2bc11bbad4c33930efb5a940c3506 (127.15.102.194:44163)
I20260620 18:47:35.963337 30456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4675b4ce9df04c35b3a45256990316f9" candidate_uuid: "86b120ef400f4152a059a9d5189e7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a2bc11bbad4c33930efb5a940c3506"
I20260620 18:47:35.963357 30324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4675b4ce9df04c35b3a45256990316f9" candidate_uuid: "86b120ef400f4152a059a9d5189e7a1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88dd7880435b43c39f624069b8025012"
I20260620 18:47:35.963544 30456 raft_consensus.cc:3037] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:35.963554 30324 raft_consensus.cc:3037] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:35.965812 30456 raft_consensus.cc:2454] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b120ef400f4152a059a9d5189e7a1c in term 1.
I20260620 18:47:35.965891 30324 raft_consensus.cc:2454] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b120ef400f4152a059a9d5189e7a1c in term 1.
I20260620 18:47:35.966264 30520 leader_election.cc:302] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a2bc11bbad4c33930efb5a940c3506, 86b120ef400f4152a059a9d5189e7a1c; no voters:
I20260620 18:47:35.966542 30655 raft_consensus.cc:2781] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:35.967012 30655 raft_consensus.cc:686] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 1 LEADER]: Becoming Leader. State: Replica: 86b120ef400f4152a059a9d5189e7a1c, State: Running, Role: LEADER
I20260620 18:47:35.967573 30655 consensus_queue.cc:237] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } }
I20260620 18:47:35.972502 30183 catalog_manager.cc:5495] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b120ef400f4152a059a9d5189e7a1c (127.15.102.195). New cstate: current_term: 1 leader_uuid: "86b120ef400f4152a059a9d5189e7a1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86b120ef400f4152a059a9d5189e7a1c" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44537 } health_report { overall_health: HEALTHY } } }
I20260620 18:47:36.017167 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:36.011938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.15.102.195:44537)
0620 18:47:36.012164 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:36.012169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:36.012187 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:47:36.012272 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:36.012276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:36.012349 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:36.012504 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:36.012879 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:36.013605 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:36.015015 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:36.015466 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:36.015602 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:36.016108 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:36.016601 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:36.016773 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:36.016824 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:47:36.016904 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:47:36.025638 30324 raft_consensus.cc:1261] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 1 FOLLOWER]: Refusing update from remote peer 86b120ef400f4152a059a9d5189e7a1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:36.025635 30456 raft_consensus.cc:1261] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 1 FOLLOWER]: Refusing update from remote peer 86b120ef400f4152a059a9d5189e7a1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:36.026799 30658 consensus_queue.cc:1035] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a2bc11bbad4c33930efb5a940c3506" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:36.027138 30655 consensus_queue.cc:1035] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "88dd7880435b43c39f624069b8025012" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 41367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:36.038565 30662 mvcc.cc:205] Tried to move back new op lower bound from 7298995224669331456 to 7298995224447131648. Current Snapshot: MvccSnapshot[applied={T|T < 7298995224669331456}]
I20260620 18:47:36.040558 30661 mvcc.cc:205] Tried to move back new op lower bound from 7298995224669331456 to 7298995224447131648. Current Snapshot: MvccSnapshot[applied={T|T < 7298995224669331456}]
I20260620 18:47:36.051770 30182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:47:36.052085 30182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:47:36.056057 30182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Sending DeleteTablet for 3 replicas of tablet 4675b4ce9df04c35b3a45256990316f9
I20260620 18:47:36.057241 30436 tablet_service.cc:1514] Processing DeleteTablet for tablet 4675b4ce9df04c35b3a45256990316f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:36 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:56444
I20260620 18:47:36.057241 30304 tablet_service.cc:1514] Processing DeleteTablet for tablet 4675b4ce9df04c35b3a45256990316f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:36 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:44332
I20260620 18:47:36.057235 30567 tablet_service.cc:1514] Processing DeleteTablet for tablet 4675b4ce9df04c35b3a45256990316f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:36 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:49090
I20260620 18:47:36.059075 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 30221
I20260620 18:47:36.063937 30671 tablet_replica.cc:330] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: stopping tablet replica
I20260620 18:47:36.064677 30671 raft_consensus.cc:2229] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:47:36.065210 30671 raft_consensus.cc:2258] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:36.069620 30671 ts_tablet_manager.cc:1905] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:36.073194 30670 tablet_replica.cc:330] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: stopping tablet replica
I20260620 18:47:36.073668 30670 raft_consensus.cc:2229] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:47:36.074294 30670 raft_consensus.cc:2258] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:36.075270 30672 tablet_replica.cc:330] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: stopping tablet replica
I20260620 18:47:36.076260 30672 raft_consensus.cc:2229] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:47:36.076768 30670 ts_tablet_manager.cc:1905] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:36.076957 30672 raft_consensus.cc:2258] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:36.077394 30671 ts_tablet_manager.cc:1918] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:36.077562 30671 log.cc:1196] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/4675b4ce9df04c35b3a45256990316f9
I20260620 18:47:36.078212 30671 ts_tablet_manager.cc:1939] T 4675b4ce9df04c35b3a45256990316f9 P 88dd7880435b43c39f624069b8025012: Deleting consensus metadata
I20260620 18:47:36.079162 30672 ts_tablet_manager.cc:1905] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:36.079957 30170 catalog_manager.cc:4841] TS 88dd7880435b43c39f624069b8025012 (127.15.102.193:41367): tablet 4675b4ce9df04c35b3a45256990316f9 (table test-table [id=94bbb5beb52b4175a67977299c18e9bb]) successfully deleted
I20260620 18:47:36.083571 30670 ts_tablet_manager.cc:1918] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:36.083745 30670 log.cc:1196] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/4675b4ce9df04c35b3a45256990316f9
I20260620 18:47:36.084313 30670 ts_tablet_manager.cc:1939] T 4675b4ce9df04c35b3a45256990316f9 P 44a2bc11bbad4c33930efb5a940c3506: Deleting consensus metadata
I20260620 18:47:36.084452 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:36.060040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.15.102.193:41367)
0620 18:47:36.060259 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:36.060267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:36.060294 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:47:36.060372 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:36.060380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:36.060482 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:36.060616 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:36.062332 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:36.063837 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:36.065883 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:36.066762 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:36.066918 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:36.070552 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:36.070568 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:36.070587 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:36.070628 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:36.074713 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:36.075835 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:36.075844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:36.075848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:36.076140 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:36.077591 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:36.077597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:36.077601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:36.078105 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:36.078447 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:36.083921 (+ 5474us) server_negotiation.cc:299] Negotiation successful
0620 18:47:36.084123 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:47:36.085253 30364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:36.086452 30168 catalog_manager.cc:4841] TS 44a2bc11bbad4c33930efb5a940c3506 (127.15.102.194:44163): tablet 4675b4ce9df04c35b3a45256990316f9 (table test-table [id=94bbb5beb52b4175a67977299c18e9bb]) successfully deleted
I20260620 18:47:36.087309 30672 ts_tablet_manager.cc:1918] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:36.087525 30672 log.cc:1196] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/4675b4ce9df04c35b3a45256990316f9
I20260620 18:47:36.088109 30672 ts_tablet_manager.cc:1939] T 4675b4ce9df04c35b3a45256990316f9 P 86b120ef400f4152a059a9d5189e7a1c: Deleting consensus metadata
I20260620 18:47:36.089944 30171 catalog_manager.cc:4841] TS 86b120ef400f4152a059a9d5189e7a1c (127.15.102.195:44537): tablet 4675b4ce9df04c35b3a45256990316f9 (table test-table [id=94bbb5beb52b4175a67977299c18e9bb]) successfully deleted
I20260620 18:47:36.185913 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 30221
I20260620 18:47:36.203752 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 30373
I20260620 18:47:36.220299 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:36.204581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.15.102.194:44163)
0620 18:47:36.204759 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:36.204763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:36.204784 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:47:36.204990 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:36.204994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:36.205059 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:36.205141 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:36.206927 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:36.207830 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:36.209708 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:36.210106 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:36.210221 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:36.213418 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:36.213436 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:36.213459 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:36.213504 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:36.217404 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:36.218051 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:36.218061 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:36.218065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:36.218209 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:36.218598 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:36.218603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:36.218607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:36.219098 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:36.219430 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:36.219712 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 18:47:36.219925 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:47:36.221100 30496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:36.310600 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 30373
I20260620 18:47:36.328559 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 30504
I20260620 18:47:36.345374 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:36.329432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.15.102.195:44537)
0620 18:47:36.329626 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:36.329636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:36.329658 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:47:36.329785 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:36.329790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:36.329903 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:36.330031 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:36.331476 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:36.332749 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:36.334155 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:36.334878 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:36.335014 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:36.338018 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:36.338029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:36.338047 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:36.338095 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:36.342153 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:36.342922 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:36.342931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:36.342935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:36.343129 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:36.343656 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:36.343662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:36.343665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:36.344184 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:36.344510 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:36.344685 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 18:47:36.344939 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:47:36.346158 30627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:36.470597 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 30504
I20260620 18:47:36.489858 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 30151
I20260620 18:47:36.491065 30214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:36.605496 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 30151
I20260620 18:47:36.627463 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:34189
--webserver_interface=127.15.102.254
--webserver_port=34503
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:36.962251 30677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:36.962544 30677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:36.962771 30677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:36.971535 30677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:36.971661 30677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:36.971745 30677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:36.971820 30677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:36.971884 30677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:36.981922 30677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=34503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:36.983350 30677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:36.984732 30677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:36.996634 30683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:36.996706 30682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:36.997251 30677 server_base.cc:1034] running on GCE node
W20260620 18:47:36.997421 30685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:36.998262 30677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:36.999866 30677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:37.001099 30677 hybrid_clock.cc:648] HybridClock initialized: now 1781981257001055 us; error 57 us; skew 500 ppm
I20260620 18:47:37.001451 30677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:36 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981256, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:37.004642 30677 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:47:37.006141 30677 webserver.cc:460] Webserver started at http://127.15.102.254:34503/ using document root <none> and password file <none>
I20260620 18:47:37.006882 30677 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:37.006985 30677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:37.012578 30677 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:47:37.015588 30692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:37.016984 30677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:47:37.017122 30677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "394f633ec05c4078a6a98e8273b12c70"
format_stamp: "Formatted at 2026-06-20 18:47:30 on dist-test-slave-zfp8"
server_key: "81e4d7cb58f21d8fa9bbf3767bc2b39c"
server_key_iv: "07e4583bb4a247dc42f54a856ff27fd5"
server_key_version: "encryptionkey@0"
I20260620 18:47:37.018045 30677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:37.053954 30677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:37.058176 30677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:37.058400 30677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:37.077080 30677 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:34189
I20260620 18:47:37.077088 30743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:34189 every 8 connection(s)
I20260620 18:47:37.078619 30677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:37.080397 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 30677
I20260620 18:47:37.081202 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:41367
--local_ip_for_outbound_sockets=127.15.102.193
--tserver_master_addrs=127.15.102.254:34189
--webserver_port=35453
--webserver_interface=127.15.102.193
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:37.086258 30744 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:47:37.090852 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Bootstrap starting.
I20260620 18:47:37.121654 30744 log.cc:824] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:37.138329 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:47:37.138901 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70: Bootstrap complete.
I20260620 18:47:37.148166 30744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:37.149096 30744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394f633ec05c4078a6a98e8273b12c70, State: Initialized, Role: FOLLOWER
I20260620 18:47:37.149747 30744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:37.149998 30744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:37.150123 30744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:37.150309 30744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:47:37.154420 30744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:37.154942 30744 leader_election.cc:302] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 394f633ec05c4078a6a98e8273b12c70; no voters:
I20260620 18:47:37.155711 30744 leader_election.cc:288] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:47:37.155856 30748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:47:37.157055 30748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [term 3 LEADER]: Becoming Leader. State: Replica: 394f633ec05c4078a6a98e8273b12c70, State: Running, Role: LEADER
I20260620 18:47:37.157655 30748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } }
I20260620 18:47:37.157967 30744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:37.160283 30749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "394f633ec05c4078a6a98e8273b12c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } } }
I20260620 18:47:37.160748 30749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:37.165535 30750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394f633ec05c4078a6a98e8273b12c70. Latest consensus state: current_term: 3 leader_uuid: "394f633ec05c4078a6a98e8273b12c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f633ec05c4078a6a98e8273b12c70" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 34189 } } }
I20260620 18:47:37.166493 30750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394f633ec05c4078a6a98e8273b12c70 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:37.168502 30754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:37.178117 30754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:37.179726 30754 catalog_manager.cc:1259] Loaded cluster ID: 557950285866474297c662a6d5afbd56
I20260620 18:47:37.179896 30754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 18:47:37.181921 30754 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f68044c19ad google::LogMessage::Fail() at ??:0
@ 0x7f68044c488a google::LogMessage::SendToLog() at ??:0
@ 0x7f68044c14de google::LogMessage::Flush() at ??:0
@ 0x7f68044c2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6819ef4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6819ef1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260620 18:47:37.203789 30766 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f6819f79610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6819f79439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f680710d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f680711091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6807110799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f68070ebeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f680a3c56db start_thread at ??:0
@ 0x7f680335b61f clone at ??:0
W20260620 18:47:37.483150 30746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:37.483515 30746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:37.483682 30746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:37.491154 30746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:37.491354 30746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:37.491557 30746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:47:37.501749 30746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:41367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=35453
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:37.503163 30746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:37.504581 30746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:37.518673 30785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:37.518714 30786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:37.518625 30788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:37.519914 30746 server_base.cc:1034] running on GCE node
I20260620 18:47:37.520623 30746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:37.521716 30746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:37.522984 30746 hybrid_clock.cc:648] HybridClock initialized: now 1781981257522930 us; error 62 us; skew 500 ppm
I20260620 18:47:37.523417 30746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:37 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981257, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:37.532274 30746 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:47:37.533802 30746 webserver.cc:460] Webserver started at http://127.15.102.193:35453/ using document root <none> and password file <none>
I20260620 18:47:37.534533 30746 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:37.534683 30746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:37.540639 30746 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:47:37.543556 30795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:37.544984 30746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:47:37.545199 30746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "88dd7880435b43c39f624069b8025012"
format_stamp: "Formatted at 2026-06-20 18:47:31 on dist-test-slave-zfp8"
server_key: "a85d9409411090332545321e7169c8a8"
server_key_iv: "7e530f3589941b6a51842cb0f1fdf149"
server_key_version: "encryptionkey@0"
I20260620 18:47:37.546233 30746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:37.563344 30746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:37.567425 30746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:37.567715 30746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:37.568781 30746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:37.570158 30746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:37.570288 30746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:37.570434 30746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:37.570530 30746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:37.604615 30746 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:41367
I20260620 18:47:37.604672 30907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:41367 every 8 connection(s)
I20260620 18:47:37.606118 30746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:47:37.609995 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 30746
I20260620 18:47:37.610661 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:44163
--local_ip_for_outbound_sockets=127.15.102.194
--tserver_master_addrs=127.15.102.254:34189
--webserver_port=33611
--webserver_interface=127.15.102.194
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:37.611145 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:37.609107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:41157)
0620 18:47:37.609885 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:37.610136 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":339,"threads_started":1}
W20260620 18:47:37.611809 30908 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:34189 (0 consecutive failures): Network error: Failed to ping master at 127.15.102.254:34189: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
I20260620 18:47:37.612710 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:37.612312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:35269)
0620 18:47:37.612468 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:37.612557 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:47:37.613674 30910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:37.613228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:54479)
0620 18:47:37.613415 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:37.613517 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:47:37.614046 30908 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:47:37.937633 30911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:37.937911 30911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:37.938086 30911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:37.945179 30911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:37.945291 30911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:37.945485 30911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:47:37.955515 30911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:44163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=33611
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:37.956833 30911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:37.958163 30911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:37.972455 30916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:37.972430 30917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:37.972664 30919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:37.973347 30911 server_base.cc:1034] running on GCE node
I20260620 18:47:37.973982 30911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:37.975024 30911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:37.976274 30911 hybrid_clock.cc:648] HybridClock initialized: now 1781981257976220 us; error 69 us; skew 500 ppm
I20260620 18:47:37.976612 30911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:37 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981257, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:37.979768 30911 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:47:37.981237 30911 webserver.cc:460] Webserver started at http://127.15.102.194:33611/ using document root <none> and password file <none>
I20260620 18:47:37.981963 30911 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:37.982110 30911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:37.987505 30911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:47:37.990211 30926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:37.991637 30911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:47:37.991834 30911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "44a2bc11bbad4c33930efb5a940c3506"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "1ba05c92acdaad88ea85d94f51a69477"
server_key_iv: "c63e6e684f637c955808bdfe9a802f95"
server_key_version: "encryptionkey@0"
I20260620 18:47:37.992822 30911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:38.009148 30911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:38.013381 30911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:38.013690 30911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:38.014750 30911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:38.016275 30911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:38.016372 30911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:38.016453 30911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:38.016513 30911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:38.050246 30911 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:44163
I20260620 18:47:38.050318 31038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:44163 every 8 connection(s)
I20260620 18:47:38.051882 30911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:47:38.053439 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 30911
I20260620 18:47:38.054092 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:44537
--local_ip_for_outbound_sockets=127.15.102.195
--tserver_master_addrs=127.15.102.254:34189
--webserver_port=36973
--webserver_interface=127.15.102.195
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:46421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:47:38.058897 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:38.056526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:34025)
0620 18:47:38.057416 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:38.057690 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":318,"threads_started":1}
W20260620 18:47:38.059607 31039 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:34189 (0 consecutive failures): Network error: Failed to ping master at 127.15.102.254:34189: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
I20260620 18:47:38.060549 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:38.060204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:41279)
0620 18:47:38.060375 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:38.060443 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 18:47:38.061437 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:38.061143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:38831)
0620 18:47:38.061272 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:38.061340 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:47:38.061703 31039 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:47:38.386598 31042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:38.386972 31042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:38.387133 31042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:38.394492 31042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:38.394605 31042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:38.394793 31042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:47:38.404955 31042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:44537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=36973
--tserver_master_addrs=127.15.102.254:34189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:38.406320 31042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:38.407759 31042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:38.421240 31048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:38.421255 31050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:38.421504 31047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:38.421958 31042 server_base.cc:1034] running on GCE node
I20260620 18:47:38.422590 31042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:38.423501 31042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:38.424731 31042 hybrid_clock.cc:648] HybridClock initialized: now 1781981258424678 us; error 46 us; skew 500 ppm
I20260620 18:47:38.425048 31042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:38 dist-test-slave-zfp8 krb5kdc[29598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981258, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:38.428130 31042 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:47:38.429503 31042 webserver.cc:460] Webserver started at http://127.15.102.195:36973/ using document root <none> and password file <none>
I20260620 18:47:38.430195 31042 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:38.430341 31042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:38.435600 31042 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:47:38.438225 31057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:38.439586 31042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:47:38.439759 31042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "86b120ef400f4152a059a9d5189e7a1c"
format_stamp: "Formatted at 2026-06-20 18:47:32 on dist-test-slave-zfp8"
server_key: "ca4449eee88d5050348af4828b5ff4ee"
server_key_iv: "919363098dadf269a4f4299d7613c304"
server_key_version: "encryptionkey@0"
I20260620 18:47:38.440721 31042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:38.458626 31042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:38.463570 31042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:38.463860 31042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:38.464828 31042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:38.466280 31042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:38.466378 31042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:38.466547 31042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:38.466650 31042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:38.498908 31042 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:44537
I20260620 18:47:38.498991 31169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:44537 every 8 connection(s)
I20260620 18:47:38.500434 31042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:47:38.505425 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:38.503668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:41789)
0620 18:47:38.504378 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:38.504587 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":281,"threads_started":1}
W20260620 18:47:38.506042 31170 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:34189 (0 consecutive failures): Network error: Failed to ping master at 127.15.102.254:34189: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
I20260620 18:47:38.506757 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:38.506459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:49821)
0620 18:47:38.506590 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:38.506652 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:47:38.507572 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:38.507220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:36767)
0620 18:47:38.507387 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:38.507468 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 18:47:38.507790 31170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:47:38.509198 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 31042
I20260620 18:47:38.615536 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:38.614581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:41667)
0620 18:47:38.615139 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:38.615313 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":310,"threads_started":1}
I20260620 18:47:39.063133 31175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:39.062294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:56791)
0620 18:47:39.062774 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:39.062905 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":231,"threads_started":1}
I20260620 18:47:39.509205 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:39.508355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:59391)
0620 18:47:39.508821 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:39.508968 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260620 18:47:39.617328 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:39.616488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:54269)
0620 18:47:39.617073 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:39.617164 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":312,"threads_started":1}
I20260620 18:47:40.064930 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:40.064173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:51591)
0620 18:47:40.064668 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:40.064757 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":232,"threads_started":1}
I20260620 18:47:40.510965 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:40.510221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:49951)
0620 18:47:40.510708 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:40.510797 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":243,"threads_started":1}
I20260620 18:47:40.619007 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:40.618228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:49457)
0620 18:47:40.618740 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:40.618838 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":299,"threads_started":1}
I20260620 18:47:41.067512 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:41.065962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:37743)
0620 18:47:41.067164 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:41.067325 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1069,"thread_start_us":951,"threads_started":1}
I20260620 18:47:41.513070 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:41.512029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:60371)
0620 18:47:41.512661 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:41.512787 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":314,"threads_started":1}
I20260620 18:47:41.620677 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:41.619936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:36893)
0620 18:47:41.620410 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:41.620492 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
I20260620 18:47:42.069479 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:42.068546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:37539)
0620 18:47:42.069132 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:42.069242 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":247,"threads_started":1}
I20260620 18:47:42.514771 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:42.514038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:48823)
0620 18:47:42.514463 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:42.514562 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":210,"threads_started":1}
I20260620 18:47:42.622505 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:42.621691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:35653)
0620 18:47:42.622224 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:42.622316 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":292,"threads_started":1}
I20260620 18:47:43.071218 31187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:43.070418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:51279)
0620 18:47:43.070952 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:43.071042 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":304,"threads_started":1}
I20260620 18:47:43.516680 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:43.515888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:43939)
0620 18:47:43.516420 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:43.516510 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":291,"threads_started":1}
I20260620 18:47:43.624465 31189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:43.623619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:42577)
0620 18:47:43.624107 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:43.624247 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":243,"threads_started":1}
I20260620 18:47:44.072882 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:44.072168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:34985)
0620 18:47:44.072630 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:44.072718 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":227,"threads_started":1}
I20260620 18:47:44.518697 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:44.517773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:35223)
0620 18:47:44.518391 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:44.518522 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":337,"threads_started":1}
I20260620 18:47:44.626295 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:44.625497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:50107)
0620 18:47:44.625997 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:44.626085 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":218,"threads_started":1}
I20260620 18:47:45.074746 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:45.073836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:41175)
0620 18:47:45.074418 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:45.074506 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":248,"threads_started":1}
I20260620 18:47:45.520637 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:45.519795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:38425)
0620 18:47:45.520327 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:45.520416 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":234,"threads_started":1}
I20260620 18:47:45.628041 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:45.627329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:58475)
0620 18:47:45.627781 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:45.627867 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
I20260620 18:47:46.076723 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:46.075974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:42263)
0620 18:47:46.076472 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:46.076555 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":247,"threads_started":1}
I20260620 18:47:46.522413 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:46.521577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:36961)
0620 18:47:46.522155 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:46.522246 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":339,"threads_started":1}
I20260620 18:47:46.629869 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:46.629052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:58327)
0620 18:47:46.629617 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:46.629699 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":302,"threads_started":1}
I20260620 18:47:47.078413 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:47.077671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:48525)
0620 18:47:47.078158 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:47.078241 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":230,"threads_started":1}
I20260620 18:47:47.524104 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:47.523338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:43125)
0620 18:47:47.523848 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:47.523932 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
I20260620 18:47:47.631733 31201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:47.630902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:51699)
0620 18:47:47.631487 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:47.631569 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":294,"threads_started":1}
I20260620 18:47:48.080154 31202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:48.079446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:57757)
0620 18:47:48.079909 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:48.079991 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":245,"threads_started":1}
I20260620 18:47:48.526140 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:48.525313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:55741)
0620 18:47:48.525859 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:48.525961 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":295,"threads_started":1}
I20260620 18:47:48.633543 31204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:48.632821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:56429)
0620 18:47:48.633258 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:48.633352 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
I20260620 18:47:49.081986 31205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:49.081209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:38377)
0620 18:47:49.081682 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:49.081781 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":235,"threads_started":1}
I20260620 18:47:49.528146 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:49.527355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:45245)
0620 18:47:49.527872 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:49.527969 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":280,"threads_started":1}
I20260620 18:47:49.635780 31207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:49.634900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:52055)
0620 18:47:49.635501 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:49.635596 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":276,"threads_started":1}
I20260620 18:47:50.083868 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:50.083001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:48227)
0620 18:47:50.083619 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:50.083704 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":374,"threads_started":1}
I20260620 18:47:50.530031 31209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:50.529249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:48113)
0620 18:47:50.529764 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:50.529858 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":270,"threads_started":1}
I20260620 18:47:50.637856 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:50.636955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:45465)
0620 18:47:50.637532 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:50.637642 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":288,"threads_started":1}
I20260620 18:47:51.086158 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:51.084971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:47893)
0620 18:47:51.085777 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:51.085937 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":431,"threads_started":1}
I20260620 18:47:51.532243 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:51.531127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:58583)
0620 18:47:51.531846 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:51.531968 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":393,"threads_started":1}
I20260620 18:47:51.639955 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:51.639031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:37291)
0620 18:47:51.639630 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:51.639729 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":338,"threads_started":1}
I20260620 18:47:52.088397 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:52.087571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:51001)
0620 18:47:52.088081 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:52.088186 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
I20260620 18:47:52.534135 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:52.533318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:46939)
0620 18:47:52.533840 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:52.533948 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":277,"threads_started":1}
I20260620 18:47:52.641872 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:52.641128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.193:40045)
0620 18:47:52.641580 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:52.641678 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":245,"threads_started":1}
I20260620 18:47:53.090726 31217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:53.089876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.194:55517)
0620 18:47:53.090410 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:53.090521 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
I20260620 18:47:53.518925 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 30746
I20260620 18:47:53.536201 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:53.535406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:34189 (local address 127.15.102.195:43385)
0620 18:47:53.535921 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:53.536019 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":276,"threads_started":1}
I20260620 18:47:53.541707 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:53.520286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.15.102.193:41367)
0620 18:47:53.521038 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:53.521059 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:53.521109 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:47:53.523586 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:53.523600 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:53.524104 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:53.524455 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:53.526037 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:53.527577 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:53.529118 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:53.529875 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:53.530032 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:53.533661 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:53.533670 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:53.533689 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:53.533740 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:53.537689 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:53.538558 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:53.538566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:53.538574 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:53.538757 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:53.539415 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:53.539419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:53.539422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:53.540057 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:53.540318 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:53.540891 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 18:47:53.541384 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":357,"threads_started":1}
I20260620 18:47:53.543467 30903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:53.633178 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 30746
I20260620 18:47:53.652383 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 30911
I20260620 18:47:53.671953 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:53.653549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.15.102.194:44163)
0620 18:47:53.654142 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:53.654157 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:53.654200 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:47:53.656051 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:53.656064 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:53.656560 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:53.656904 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:53.658364 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:53.659905 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:53.661242 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:53.661994 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:53.662155 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:53.664999 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:53.665007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:53.665024 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:53.665066 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:53.668361 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:53.669153 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:53.669165 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:53.669175 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:53.669391 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:53.669808 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:53.669813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:53.669815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:53.670436 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:53.670760 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:53.671061 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 18:47:53.671620 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":287,"threads_started":1}
I20260620 18:47:53.673471 31034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:53.758612 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 30911
I20260620 18:47:53.777441 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 31042
I20260620 18:47:53.796216 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:53.778576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46900 (local address 127.15.102.195:44537)
0620 18:47:53.779170 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:53.779184 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:53.779291 (+ 107us) server_negotiation.cc:407] Connection header received
0620 18:47:53.780958 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:53.780968 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:53.781421 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:53.781801 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:53.783359 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:53.784603 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:53.785930 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:53.786631 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:53.786858 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:53.789483 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:53.789492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:53.789509 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:53.789549 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:53.792665 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:53.793407 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:53.793416 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:53.793427 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:53.793602 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:53.793997 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:53.794003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:53.794007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:53.794584 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:53.794913 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:53.795422 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 18:47:53.795914 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":275,"threads_started":1}
I20260620 18:47:53.797676 31165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:53.881902 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 31042
2026-06-20T18:47:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24807 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:47:53 dist-test-slave-zfp8 krb5kdc[31227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:47:53 dist-test-slave-zfp8 krb5kdc[31227](info): set up 2 sockets
Jun 20 18:47:53 dist-test-slave-zfp8 krb5kdc[31227](info): commencing operation
krb5kdc: starting...
W20260620 18:47:55.023977 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:47:55 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:47:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:47:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:47:55.279560 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36635
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:55.641148 31240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:55.641471 31240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:55.641561 31240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:55.648988 31240 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:47:55.649119 31240 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:47:55.649158 31240 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:47:55.649204 31240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:47:55.649257 31240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:55.649327 31240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:55.649372 31240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:47:55.649418 31240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:47:55.659967 31240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:36635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:55.661319 31240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:55.662698 31240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:55.675974 31246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:55.675894 31248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:55.675977 31245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:55.677012 31240 server_base.cc:1034] running on GCE node
I20260620 18:47:55.677874 31240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:55.679514 31240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:55.680815 31240 hybrid_clock.cc:648] HybridClock initialized: now 1781981275680719 us; error 53 us; skew 500 ppm
I20260620 18:47:55.681281 31240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:55 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981275, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:55.685092 31240 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:47:55.686966 31240 webserver.cc:460] Webserver started at http://127.15.102.254:39073/ using document root <none> and password file <none>
I20260620 18:47:55.687922 31240 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:55.688076 31240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:55.688452 31240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:55.692180 31240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "ef7577c548e849feae1e4761109ff3f9"
format_stamp: "Formatted at 2026-06-20 18:47:55 on dist-test-slave-zfp8"
server_key: "4d48c90a8cc16bf3c743991de4e5b91d"
server_key_iv: "536978fe22a6134aea03cb1d8288d024"
server_key_version: "encryptionkey@0"
I20260620 18:47:55.693166 31240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "ef7577c548e849feae1e4761109ff3f9"
format_stamp: "Formatted at 2026-06-20 18:47:55 on dist-test-slave-zfp8"
server_key: "4d48c90a8cc16bf3c743991de4e5b91d"
server_key_iv: "536978fe22a6134aea03cb1d8288d024"
server_key_version: "encryptionkey@0"
I20260620 18:47:55.699429 31240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 18:47:55.703759 31255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:55.705536 31240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 18:47:55.705811 31240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "ef7577c548e849feae1e4761109ff3f9"
format_stamp: "Formatted at 2026-06-20 18:47:55 on dist-test-slave-zfp8"
server_key: "4d48c90a8cc16bf3c743991de4e5b91d"
server_key_iv: "536978fe22a6134aea03cb1d8288d024"
server_key_version: "encryptionkey@0"
I20260620 18:47:55.706091 31240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:55.727880 31240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:55.732378 31240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:55.732688 31240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:55.752660 31240 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:36635
I20260620 18:47:55.752625 31306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:36635 every 8 connection(s)
I20260620 18:47:55.754130 31240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:47:55.755599 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 31240
I20260620 18:47:55.755836 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:47:55.756515 15771 external_mini_cluster.cc:1428] Setting key 6762e320a6eb41d9ed69b337cecf9337
I20260620 18:47:55.758955 31307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:47:55 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:55.773448 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Bootstrap starting.
I20260620 18:47:55.778055 31307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:55.779498 31307 log.cc:824] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:55.783352 31307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: No bootstrap required, opened a new log
I20260620 18:47:55.783811 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:55.758713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.15.102.254:36635)
0620 18:47:55.759579 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:55.759597 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:55.759654 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:47:55.762071 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:55.762103 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:55.762725 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:55.763275 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:55.764803 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:55.766937 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:55.768455 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:55.769593 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:55.769796 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:55.774458 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:55.774469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:55.774492 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:55.774548 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:55.778786 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:55.779684 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:55.779694 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:55.779705 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:55.779934 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:55.780429 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:55.780433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:55.780436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:55.781128 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:55.781445 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:55.781853 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 18:47:55.782351 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":353,"threads_started":1}
I20260620 18:47:55.792192 31307 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:47:55.792603 31307 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:55.792680 31307 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7577c548e849feae1e4761109ff3f9, State: Initialized, Role: FOLLOWER
I20260620 18:47:55.793282 31307 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:47:55.793500 31307 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:47:55.793606 31307 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:47:55.793743 31307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:55.796311 31307 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:47:55.796829 31307 leader_election.cc:302] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef7577c548e849feae1e4761109ff3f9; no voters:
I20260620 18:47:55.797377 31307 leader_election.cc:288] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:47:55.797535 31312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:55.798228 31312 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 1 LEADER]: Becoming Leader. State: Replica: ef7577c548e849feae1e4761109ff3f9, State: Running, Role: LEADER
I20260620 18:47:55.798827 31312 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:47:55.799677 31307 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:47:55.801756 31314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef7577c548e849feae1e4761109ff3f9. Latest consensus state: current_term: 1 leader_uuid: "ef7577c548e849feae1e4761109ff3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } } }
I20260620 18:47:55.802086 31314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:55.803004 31317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:47:55.803574 31313 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef7577c548e849feae1e4761109ff3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } } }
I20260620 18:47:55.804100 31313 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:47:55.810128 31317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:47:55.819005 31317 catalog_manager.cc:1347] Generated new cluster ID: 8eef1b28713947418e03d5d7a4291cf5
I20260620 18:47:55.819141 31317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:47:55.838192 31317 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:47:55.840268 31317 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:47:55.850971 31317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Generated new TSK 0
I20260620 18:47:55.852002 31317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:47:55.923532 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36635
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:56.278412 31333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:56.278774 31333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:56.278935 31333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:56.286972 31333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:56.287132 31333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:56.287410 31333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:47:56.298308 31333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:56.299997 31333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:56.301707 31333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:56.316937 31339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:56.316946 31338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:56.317060 31341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:56.317919 31333 server_base.cc:1034] running on GCE node
I20260620 18:47:56.318656 31333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:56.319905 31333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:56.321170 31333 hybrid_clock.cc:648] HybridClock initialized: now 1781981276321094 us; error 71 us; skew 500 ppm
I20260620 18:47:56.321552 31333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:56 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981276, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:56.325367 31333 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:47:56.327028 31333 webserver.cc:460] Webserver started at http://127.15.102.193:44525/ using document root <none> and password file <none>
I20260620 18:47:56.327888 31333 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:56.328019 31333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:56.328336 31333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:56.331995 31333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "f39e9cb8e7624404ab71f828fbb64bef"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "21edfe41c9a3db5a3419028764f43a7b"
server_key_iv: "3d04152e23d2e92f969e4472a13cecdd"
server_key_version: "encryptionkey@0"
I20260620 18:47:56.333395 31333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "f39e9cb8e7624404ab71f828fbb64bef"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "21edfe41c9a3db5a3419028764f43a7b"
server_key_iv: "3d04152e23d2e92f969e4472a13cecdd"
server_key_version: "encryptionkey@0"
I20260620 18:47:56.340054 31333 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 18:47:56.344694 31348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:56.346506 31333 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:47:56.346796 31333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "f39e9cb8e7624404ab71f828fbb64bef"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "21edfe41c9a3db5a3419028764f43a7b"
server_key_iv: "3d04152e23d2e92f969e4472a13cecdd"
server_key_version: "encryptionkey@0"
I20260620 18:47:56.347083 31333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:56.372174 31333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:56.377362 31333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:56.377749 31333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:56.378902 31333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:56.380651 31333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:56.380744 31333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:56.380889 31333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:56.380980 31333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:56.418605 31333 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:35187
I20260620 18:47:56.418671 31460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:35187 every 8 connection(s)
I20260620 18:47:56.420192 31333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:47:56.422207 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 31333
I20260620 18:47:56.422465 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:47:56.423328 15771 external_mini_cluster.cc:1428] Setting key 0bc7d46be389f1701e3328ad4ede1051
Jun 20 18:47:56 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981276, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:56.447587 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:56.423136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:43935 (local address 127.15.102.254:36635)
0620 18:47:56.423746 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:56.423754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:56.424946 (+ 1192us) server_negotiation.cc:407] Connection header received
0620 18:47:56.427963 (+ 3017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:56.427968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:56.428041 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:56.428138 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:56.430618 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:56.431452 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:56.434512 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:56.435138 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:56.435373 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:56.439591 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:56.439601 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:56.439609 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:56.439660 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:56.443575 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:56.444499 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:56.444508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:56.444515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:56.444681 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:56.445258 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:56.445266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:56.445273 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:56.445677 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:56.445846 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:56.446708 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 18:47:56.446997 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":297,"threads_started":1}
I20260620 18:47:56.448685 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:56.423838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:43935)
0620 18:47:56.424782 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:56.424825 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:56.427596 (+ 2771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:56.428477 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:56.428493 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:56.429429 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:56.430388 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:56.430421 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:56.431654 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:56.431664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:56.434195 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:56.434225 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:56.435482 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:56.435489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:56.435756 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:56.436662 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:56.436690 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:56.439202 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:56.443774 (+ 4572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:56.443783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:56.443799 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:56.444337 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:56.444854 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:56.444858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:56.444862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:56.445125 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:56.445843 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:56.445851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:56.446268 (+ 417us) client_negotiation.cc:769] Sending connection context
0620 18:47:56.446548 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 18:47:56.447070 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":344,"threads_started":1}
I20260620 18:47:56.450649 31461 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36635
I20260620 18:47:56.451076 31461 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:56.452237 31461 heartbeater.cc:511] Master 127.15.102.254:36635 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
I20260620 18:47:56.455616 31272 ts_manager.cc:194] Registered new tserver with Master: f39e9cb8e7624404ab71f828fbb64bef (127.15.102.193:35187)
I20260620 18:47:56.457717 31272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:43935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:47:56.486011 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36635
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:56.838165 31467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:56.838483 31467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:56.838637 31467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:56.846187 31467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:56.846316 31467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:56.846498 31467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:47:56.856998 31467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:56.858398 31467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:56.859884 31467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:56.874253 31475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:56.874367 31472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:56.874334 31473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:56.875362 31467 server_base.cc:1034] running on GCE node
I20260620 18:47:56.876063 31467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:56.877118 31467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:56.878361 31467 hybrid_clock.cc:648] HybridClock initialized: now 1781981276878311 us; error 51 us; skew 500 ppm
I20260620 18:47:56.878738 31467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:47:56 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981276, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:56.885460 31467 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:47:56.887194 31467 webserver.cc:460] Webserver started at http://127.15.102.194:35451/ using document root <none> and password file <none>
I20260620 18:47:56.888069 31467 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:56.888185 31467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:56.888470 31467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:56.891863 31467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "afe52d9462fd41adb277761ade3a162b"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "af934a0413ec5da09c0ec61b3754ccb5"
server_key_iv: "ae2bf23c00f2d0309e017f58fd6c1194"
server_key_version: "encryptionkey@0"
I20260620 18:47:56.892678 31467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "afe52d9462fd41adb277761ade3a162b"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "af934a0413ec5da09c0ec61b3754ccb5"
server_key_iv: "ae2bf23c00f2d0309e017f58fd6c1194"
server_key_version: "encryptionkey@0"
I20260620 18:47:56.898463 31467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:47:56.902818 31482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:56.904588 31467 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 18:47:56.904793 31467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "afe52d9462fd41adb277761ade3a162b"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "af934a0413ec5da09c0ec61b3754ccb5"
server_key_iv: "ae2bf23c00f2d0309e017f58fd6c1194"
server_key_version: "encryptionkey@0"
I20260620 18:47:56.905012 31467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:56.942112 31467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:56.946949 31467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:56.947330 31467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:56.948457 31467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:56.950039 31467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:56.950124 31467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:56.950294 31467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:56.950361 31467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:56.984555 31467 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:36563
I20260620 18:47:56.984617 31594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:36563 every 8 connection(s)
I20260620 18:47:56.985944 31467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:47:56.993170 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 31467
I20260620 18:47:56.993391 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:47:56.993865 15771 external_mini_cluster.cc:1428] Setting key 85b9602e39c6778ab624ec311d7ee69f
Jun 20 18:47:56 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981276, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:57.009454 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:56.988743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:55461 (local address 127.15.102.254:36635)
0620 18:47:56.989190 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:56.989195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:56.990305 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 18:47:56.992371 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:56.992377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:56.992480 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:56.992607 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:56.994489 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:56.995437 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:56.997208 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:56.998061 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:56.998234 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.002159 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:57.002167 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:57.002175 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:57.002228 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:57.005448 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:57.006592 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:57.006597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:57.006599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:57.006754 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:57.007343 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:57.007347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:57.007349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:57.007622 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:57.007781 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.008837 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.009088 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260620 18:47:57.011067 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:56.989388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:55461)
0620 18:47:56.990117 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:56.990158 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:56.992048 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:56.992902 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:56.992911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:56.993580 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:56.994276 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:56.994308 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:56.995604 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:56.995608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:56.997026 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:56.997042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:56.998315 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:56.998319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:56.998543 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:56.999357 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:56.999382 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:57.001840 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:57.005648 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:57.005659 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:57.005683 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:57.006457 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:57.006876 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:57.006885 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:57.006892 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:57.007195 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:57.007792 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:57.007805 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:57.008353 (+ 548us) client_negotiation.cc:769] Sending connection context
0620 18:47:57.008683 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 18:47:57.009338 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":290,"threads_started":1}
I20260620 18:47:57.012697 31595 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36635
I20260620 18:47:57.013034 31595 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:57.013832 31595 heartbeater.cc:511] Master 127.15.102.254:36635 requested a full tablet report, sending...
I20260620 18:47:57.015440 31272 ts_manager.cc:194] Registered new tserver with Master: afe52d9462fd41adb277761ade3a162b (127.15.102.194:36563)
I20260620 18:47:57.016304 31272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:55461
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:47:57.051180 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36635
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:47:57.411825 31601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:47:57.412169 31601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:47:57.412338 31601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:47:57.421111 31601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:47:57.421268 31601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:47:57.421437 31601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:47:57.432853 31601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:47:57.434159 31601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:47:57.435611 31601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:47:57.449717 31606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:57.449774 31607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:47:57.449671 31609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:47:57.456611 31601 server_base.cc:1034] running on GCE node
I20260620 18:47:57.457523 31601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:47:57.459523 31601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:47:57.460827 31601 hybrid_clock.cc:648] HybridClock initialized: now 1781981277460703 us; error 122 us; skew 500 ppm
I20260620 18:47:57.461330 31601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:47:57.461854 31461 heartbeater.cc:503] Master 127.15.102.254:36635 was elected leader, sending a full tablet report...
Jun 20 18:47:57 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981277, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:47:57.466050 31601 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:47:57.468621 31601 webserver.cc:460] Webserver started at http://127.15.102.195:36305/ using document root <none> and password file <none>
I20260620 18:47:57.469915 31601 fs_manager.cc:359] Metadata directory not provided
I20260620 18:47:57.470120 31601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:47:57.470574 31601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:47:57.476224 31601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "b004b3b50aa4474eb66cca64976a3a4b"
format_stamp: "Formatted at 2026-06-20 18:47:57 on dist-test-slave-zfp8"
server_key: "8119525566168b1ea84b47668f347b43"
server_key_iv: "422280abb9f1912e5b9b54b0179b9210"
server_key_version: "encryptionkey@0"
I20260620 18:47:57.477497 31601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "b004b3b50aa4474eb66cca64976a3a4b"
format_stamp: "Formatted at 2026-06-20 18:47:57 on dist-test-slave-zfp8"
server_key: "8119525566168b1ea84b47668f347b43"
server_key_iv: "422280abb9f1912e5b9b54b0179b9210"
server_key_version: "encryptionkey@0"
I20260620 18:47:57.485800 31601 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260620 18:47:57.490377 31616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:57.492076 31601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:47:57.492281 31601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "b004b3b50aa4474eb66cca64976a3a4b"
format_stamp: "Formatted at 2026-06-20 18:47:57 on dist-test-slave-zfp8"
server_key: "8119525566168b1ea84b47668f347b43"
server_key_iv: "422280abb9f1912e5b9b54b0179b9210"
server_key_version: "encryptionkey@0"
I20260620 18:47:57.492506 31601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:47:57.514168 31601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:47:57.518967 31601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:47:57.519352 31601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:47:57.520375 31601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:47:57.521895 31601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:47:57.521971 31601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:57.522138 31601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:47:57.522188 31601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:47:57.556676 31601 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:42835
I20260620 18:47:57.556708 31728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:42835 every 8 connection(s)
I20260620 18:47:57.558066 31601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:47:57.561846 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 31601
I20260620 18:47:57.562064 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:47:57.562608 15771 external_mini_cluster.cc:1428] Setting key ab33787f4c3ca13482616d4ca51e5169
Jun 20 18:47:57 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981277, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:47:57.580076 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.560812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:42231 (local address 127.15.102.254:36635)
0620 18:47:57.561299 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.561304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.562378 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 18:47:57.564726 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.564734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.564827 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.564919 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:57.566808 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.567658 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.569229 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.569627 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.569724 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.573433 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:57.573439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:57.573442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:57.573480 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:57.576483 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:57.577546 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:57.577552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:57.577555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:57.577758 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:57.578262 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:57.578266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:57.578268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:57.578663 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:57.578821 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.579587 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.579805 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":261,"threads_started":1}
I20260620 18:47:57.581074 31732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.561312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:42231)
0620 18:47:57.562182 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:57.562242 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:57.564378 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:57.565240 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:57.565254 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:57.566018 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:47:57.566631 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.566651 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.567787 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.567791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.569098 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.569110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.569837 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.569842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.570044 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.570753 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:47:57.570782 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:47:57.573154 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:47:57.576672 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:57.576685 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:57.576718 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:57.577418 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:57.577884 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:47:57.577887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:47:57.577890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:47:57.578150 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:47:57.578800 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:47:57.578808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:47:57.579169 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 18:47:57.579498 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 18:47:57.579939 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":378,"threads_started":1}
I20260620 18:47:57.582451 31729 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36635
I20260620 18:47:57.582765 31729 heartbeater.cc:464] Registering TS with master...
I20260620 18:47:57.583585 31729 heartbeater.cc:511] Master 127.15.102.254:36635 requested a full tablet report, sending...
I20260620 18:47:57.585183 31272 ts_manager.cc:194] Registered new tserver with Master: b004b3b50aa4474eb66cca64976a3a4b (127.15.102.195:42835)
I20260620 18:47:57.586086 31272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:42231
I20260620 18:47:57.591143 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:47:57.610899 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.595725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.15.102.254:36635)
0620 18:47:57.595927 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.595934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.596434 (+ 500us) server_negotiation.cc:407] Connection header received
0620 18:47:57.596600 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.596605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.596703 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.596806 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:57.598430 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.599193 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.600714 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.601192 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.601290 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.604272 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:57.604279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:57.604283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:57.604330 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:57.607825 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:57.608644 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:57.608649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:57.608652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:57.608810 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:57.609216 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:57.609220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:57.609222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:57.609700 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:57.609979 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.610288 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.610514 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:47:57.615420 31272 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:47:57.622372 31272 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:47:57.663537 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.636320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.15.102.194:36563)
0620 18:47:57.653032 (+ 16712us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.653048 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.653106 (+ 58us) server_negotiation.cc:407] Connection header received
0620 18:47:57.653276 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.653290 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.653740 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.655228 (+ 1488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:57.655522 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.657341 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.660815 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.662059 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.662282 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.662511 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.662889 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.663084 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16403,"thread_start_us":17678,"threads_started":1}
I20260620 18:47:57.663772 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.636091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:36563 (local address 127.0.0.1:35398)
0620 18:47:57.642436 (+ 6345us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:57.642515 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:57.642782 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:57.654431 (+ 11649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:57.654441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:57.654494 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:57.654961 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.654975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.657499 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.657505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.660611 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.660630 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.662318 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.662330 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.662635 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.662679 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:47:57.662838 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 18:47:57.663082 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":104,"thread_start_us":722,"threads_started":1}
I20260620 18:47:57.665668 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.637126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:35187 (local address 127.0.0.1:51874)
0620 18:47:57.648882 (+ 11756us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:57.648918 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:57.649178 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:57.655408 (+ 6230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:57.655417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:57.655461 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:57.655989 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.656003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.658837 (+ 2834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.658843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.660983 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.660994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.664680 (+ 3686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.664689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.664885 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.664924 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:47:57.665039 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 18:47:57.665198 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11495,"mutex_wait_us":46,"thread_start_us":11215,"threads_started":1}
I20260620 18:47:57.665702 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.637586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.15.102.193:35187)
0620 18:47:57.654629 (+ 17043us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.654638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.654672 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:47:57.654782 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.654795 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.655094 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.655380 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:57.656216 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.658635 (+ 2419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.661155 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.664428 (+ 3273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.664658 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.664852 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.665117 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.665330 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16822,"thread_start_us":17988,"threads_started":1}
I20260620 18:47:57.666973 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.637845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:42835 (local address 127.0.0.1:35358)
0620 18:47:57.656988 (+ 19143us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:57.657013 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:57.657137 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:57.658261 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:57.658267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:57.658297 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:57.658880 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.658895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.661292 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.661298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.663667 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.663682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.666140 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.666145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.666271 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.666295 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:47:57.666416 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 18:47:57.666562 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18898,"mutex_wait_us":29,"thread_start_us":18714,"threads_started":1}
I20260620 18:47:57.667078 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.638500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.15.102.195:42835)
0620 18:47:57.656388 (+ 17888us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.656397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.657275 (+ 878us) server_negotiation.cc:407] Connection header received
0620 18:47:57.657397 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.657409 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.657712 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.657929 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:57.659196 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.661111 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.663883 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.665562 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.665793 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.665997 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.666479 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.666686 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17694,"thread_start_us":17488,"threads_started":1}
I20260620 18:47:57.668957 31530 tablet_service.cc:1467] Processing CreateTablet for tablet 1f00ec84dab34520a02d33fc2771535e (DEFAULT_TABLE table=test-table [id=6d8255b2f6e94ef4a9daa51fd3f7274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:57.670665 31530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f00ec84dab34520a02d33fc2771535e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:57.670475 31664 tablet_service.cc:1467] Processing CreateTablet for tablet 1f00ec84dab34520a02d33fc2771535e (DEFAULT_TABLE table=test-table [id=6d8255b2f6e94ef4a9daa51fd3f7274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:57.670470 31396 tablet_service.cc:1467] Processing CreateTablet for tablet 1f00ec84dab34520a02d33fc2771535e (DEFAULT_TABLE table=test-table [id=6d8255b2f6e94ef4a9daa51fd3f7274a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:47:57.672051 31396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f00ec84dab34520a02d33fc2771535e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:57.672049 31664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f00ec84dab34520a02d33fc2771535e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:47:57.685166 31748 tablet_bootstrap.cc:492] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Bootstrap starting.
I20260620 18:47:57.687964 31749 tablet_bootstrap.cc:492] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Bootstrap starting.
I20260620 18:47:57.689913 31750 tablet_bootstrap.cc:492] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: Bootstrap starting.
I20260620 18:47:57.691642 31748 tablet_bootstrap.cc:654] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:57.691880 31749 tablet_bootstrap.cc:654] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:57.693123 31750 tablet_bootstrap.cc:654] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: Neither blocks nor log segments found. Creating new log.
I20260620 18:47:57.693248 31748 log.cc:824] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:57.693248 31749 log.cc:824] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:57.694439 31750 log.cc:824] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: Log is configured to *not* fsync() on all Append() calls
I20260620 18:47:57.696910 31749 tablet_bootstrap.cc:492] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: No bootstrap required, opened a new log
I20260620 18:47:57.697261 31749 ts_tablet_manager.cc:1397] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 18:47:57.697614 31748 tablet_bootstrap.cc:492] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: No bootstrap required, opened a new log
I20260620 18:47:57.697610 31750 tablet_bootstrap.cc:492] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: No bootstrap required, opened a new log
I20260620 18:47:57.697901 31748 ts_tablet_manager.cc:1397] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260620 18:47:57.697897 31750 ts_tablet_manager.cc:1397] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 18:47:57.705900 31749 raft_consensus.cc:348] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
I20260620 18:47:57.706393 31749 raft_consensus.cc:374] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:57.706496 31749 raft_consensus.cc:729] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe52d9462fd41adb277761ade3a162b, State: Initialized, Role: FOLLOWER
I20260620 18:47:57.707311 31749 consensus_queue.cc:260] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
I20260620 18:47:57.708595 31595 heartbeater.cc:503] Master 127.15.102.254:36635 was elected leader, sending a full tablet report...
I20260620 18:47:57.709420 31749 ts_tablet_manager.cc:1428] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20260620 18:47:57.709975 31750 raft_consensus.cc:348] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
I20260620 18:47:57.710503 31750 raft_consensus.cc:374] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:57.710194 31748 raft_consensus.cc:348] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
I20260620 18:47:57.710623 31750 raft_consensus.cc:729] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39e9cb8e7624404ab71f828fbb64bef, State: Initialized, Role: FOLLOWER
I20260620 18:47:57.710780 31748 raft_consensus.cc:374] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:47:57.710908 31748 raft_consensus.cc:729] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b004b3b50aa4474eb66cca64976a3a4b, State: Initialized, Role: FOLLOWER
I20260620 18:47:57.711370 31750 consensus_queue.cc:260] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
I20260620 18:47:57.711508 31748 consensus_queue.cc:260] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
I20260620 18:47:57.713667 31750 ts_tablet_manager.cc:1428] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260620 18:47:57.716188 31729 heartbeater.cc:503] Master 127.15.102.254:36635 was elected leader, sending a full tablet report...
I20260620 18:47:57.717033 31748 ts_tablet_manager.cc:1428] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
W20260620 18:47:57.738142 31596 tablet.cc:2369] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:57.808416 31755 raft_consensus.cc:482] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:47:57.808818 31755 raft_consensus.cc:504] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
W20260620 18:47:57.809859 31730 tablet.cc:2369] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:47:57.811185 31755 leader_election.cc:288] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afe52d9462fd41adb277761ade3a162b (127.15.102.194:36563), b004b3b50aa4474eb66cca64976a3a4b (127.15.102.195:42835)
I20260620 18:47:57.820600 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.811581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:42681 (local address 127.15.102.194:36563)
0620 18:47:57.811805 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.811809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.812614 (+ 805us) server_negotiation.cc:407] Connection header received
0620 18:47:57.812870 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.812878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.812973 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.813118 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:57.813986 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.815107 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.818029 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.819449 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.819677 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.819767 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.820069 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.820194 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:47:57.820717 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.811633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:36563 (local address 127.15.102.193:42681)
0620 18:47:57.812432 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:57.812455 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:57.812616 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:57.813353 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:57.813361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:57.813399 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:57.813826 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.813841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.815311 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.815315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.817822 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.817840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.819676 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.819682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.819848 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.819901 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 18:47:57.820019 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:47:57.820192 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":37,"thread_start_us":404,"threads_started":1}
I20260620 18:47:57.820847 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.812611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:42835 (local address 127.15.102.193:58001)
0620 18:47:57.813583 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 18:47:57.813608 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:47:57.813766 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:47:57.814174 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:47:57.814178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:47:57.814199 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:47:57.814512 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.814521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.815862 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.815868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.818152 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:47:57.818163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.819943 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.819952 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:47:57.820124 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.820156 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 18:47:57.820246 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 18:47:57.820383 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":45,"thread_start_us":559,"threads_started":1}
I20260620 18:47:57.821405 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.812939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:58001 (local address 127.15.102.195:42835)
0620 18:47:57.813233 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.813240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.813706 (+ 466us) server_negotiation.cc:407] Connection header received
0620 18:47:57.813934 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.813939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.814026 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.814154 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:47:57.814682 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.815716 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.818329 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.819729 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.819889 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.819944 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.820400 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.820548 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 18:47:57.822078 31550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f00ec84dab34520a02d33fc2771535e" candidate_uuid: "f39e9cb8e7624404ab71f828fbb64bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe52d9462fd41adb277761ade3a162b" is_pre_election: true
I20260620 18:47:57.822536 31684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f00ec84dab34520a02d33fc2771535e" candidate_uuid: "f39e9cb8e7624404ab71f828fbb64bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b004b3b50aa4474eb66cca64976a3a4b" is_pre_election: true
I20260620 18:47:57.822661 31550 raft_consensus.cc:2454] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39e9cb8e7624404ab71f828fbb64bef in term 0.
I20260620 18:47:57.823107 31684 raft_consensus.cc:2454] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39e9cb8e7624404ab71f828fbb64bef in term 0.
I20260620 18:47:57.823524 31349 leader_election.cc:302] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe52d9462fd41adb277761ade3a162b, f39e9cb8e7624404ab71f828fbb64bef; no voters:
I20260620 18:47:57.824039 31755 raft_consensus.cc:2781] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:47:57.824239 31755 raft_consensus.cc:482] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:47:57.824386 31755 raft_consensus.cc:3037] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:57.827207 31755 raft_consensus.cc:504] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
I20260620 18:47:57.828114 31755 leader_election.cc:288] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [CANDIDATE]: Term 1 election: Requested vote from peers afe52d9462fd41adb277761ade3a162b (127.15.102.194:36563), b004b3b50aa4474eb66cca64976a3a4b (127.15.102.195:42835)
I20260620 18:47:57.828781 31550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f00ec84dab34520a02d33fc2771535e" candidate_uuid: "f39e9cb8e7624404ab71f828fbb64bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afe52d9462fd41adb277761ade3a162b"
I20260620 18:47:57.828948 31684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f00ec84dab34520a02d33fc2771535e" candidate_uuid: "f39e9cb8e7624404ab71f828fbb64bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b004b3b50aa4474eb66cca64976a3a4b"
I20260620 18:47:57.829003 31550 raft_consensus.cc:3037] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:57.829217 31684 raft_consensus.cc:3037] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:47:57.831437 31550 raft_consensus.cc:2454] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39e9cb8e7624404ab71f828fbb64bef in term 1.
I20260620 18:47:57.831976 31349 leader_election.cc:302] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe52d9462fd41adb277761ade3a162b, f39e9cb8e7624404ab71f828fbb64bef; no voters:
I20260620 18:47:57.832086 31684 raft_consensus.cc:2454] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39e9cb8e7624404ab71f828fbb64bef in term 1.
I20260620 18:47:57.832335 31755 raft_consensus.cc:2781] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:47:57.832935 31755 raft_consensus.cc:686] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 1 LEADER]: Becoming Leader. State: Replica: f39e9cb8e7624404ab71f828fbb64bef, State: Running, Role: LEADER
I20260620 18:47:57.833513 31755 consensus_queue.cc:237] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } }
I20260620 18:47:57.839169 31272 catalog_manager.cc:5495] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef reported cstate change: term changed from 0 to 1, leader changed from <none> to f39e9cb8e7624404ab71f828fbb64bef (127.15.102.193). New cstate: current_term: 1 leader_uuid: "f39e9cb8e7624404ab71f828fbb64bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } health_report { overall_health: HEALTHY } } }
I20260620 18:47:57.856088 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.849976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.15.102.193:35187)
0620 18:47:57.850165 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.850170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.850250 (+ 80us) server_negotiation.cc:407] Connection header received
0620 18:47:57.850462 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.850465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.850547 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.850672 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:47:57.851148 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.851964 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.853752 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.854175 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.854297 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.854846 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:47:57.855564 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:47:57.855680 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.855801 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.855876 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:47:57.865268 31550 raft_consensus.cc:1261] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 1 FOLLOWER]: Refusing update from remote peer f39e9cb8e7624404ab71f828fbb64bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:57.865271 31684 raft_consensus.cc:1261] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 1 FOLLOWER]: Refusing update from remote peer f39e9cb8e7624404ab71f828fbb64bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:47:57.866271 31755 consensus_queue.cc:1035] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:57.866631 31759 consensus_queue.cc:1035] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:47:57.879051 31762 mvcc.cc:205] Tried to move back new op lower bound from 7298995314124496896 to 7298995314011033600. Current Snapshot: MvccSnapshot[applied={T|T < 7298995314124496896}]
I20260620 18:47:57.881605 31764 mvcc.cc:205] Tried to move back new op lower bound from 7298995314124496896 to 7298995314011033600. Current Snapshot: MvccSnapshot[applied={T|T < 7298995314124496896}]
I20260620 18:47:57.894186 31271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:47:57.894500 31271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:47:57.898514 31271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Sending DeleteTablet for 3 replicas of tablet 1f00ec84dab34520a02d33fc2771535e
I20260620 18:47:57.899811 31530 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f00ec84dab34520a02d33fc2771535e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:57 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:35398
I20260620 18:47:57.899863 31664 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f00ec84dab34520a02d33fc2771535e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:57 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:35358
I20260620 18:47:57.900307 31396 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f00ec84dab34520a02d33fc2771535e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:47:57 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:51874
I20260620 18:47:57.901062 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 31333
I20260620 18:47:57.929390 31773 tablet_replica.cc:330] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: stopping tablet replica
I20260620 18:47:57.930197 31773 raft_consensus.cc:2229] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:47:57.931072 31773 raft_consensus.cc:2258] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:57.940158 31773 ts_tablet_manager.cc:1905] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:57.948913 31772 tablet_replica.cc:330] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: stopping tablet replica
I20260620 18:47:57.949206 31774 tablet_replica.cc:330] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: stopping tablet replica
I20260620 18:47:57.949555 31772 raft_consensus.cc:2229] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:47:57.949579 31774 raft_consensus.cc:2229] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:47:57.950145 31772 raft_consensus.cc:2258] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:57.950412 31774 raft_consensus.cc:2258] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:47:57.952793 31774 ts_tablet_manager.cc:1905] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:47:57 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:47:57.955933 31772 ts_tablet_manager.cc:1905] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:47:57.958144 31773 ts_tablet_manager.cc:1918] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:57.958424 31773 log.cc:1196] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/1f00ec84dab34520a02d33fc2771535e
I20260620 18:47:57.959108 31773 ts_tablet_manager.cc:1939] T 1f00ec84dab34520a02d33fc2771535e P afe52d9462fd41adb277761ade3a162b: Deleting consensus metadata
I20260620 18:47:57.961092 31774 ts_tablet_manager.cc:1918] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:57.961287 31774 log.cc:1196] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/1f00ec84dab34520a02d33fc2771535e
I20260620 18:47:57.961884 31774 ts_tablet_manager.cc:1939] T 1f00ec84dab34520a02d33fc2771535e P f39e9cb8e7624404ab71f828fbb64bef: Deleting consensus metadata
I20260620 18:47:57.961944 31256 catalog_manager.cc:4841] TS afe52d9462fd41adb277761ade3a162b (127.15.102.194:36563): tablet 1f00ec84dab34520a02d33fc2771535e (table test-table [id=6d8255b2f6e94ef4a9daa51fd3f7274a]) successfully deleted
I20260620 18:47:57.964093 31258 catalog_manager.cc:4841] TS f39e9cb8e7624404ab71f828fbb64bef (127.15.102.193:35187): tablet 1f00ec84dab34520a02d33fc2771535e (table test-table [id=6d8255b2f6e94ef4a9daa51fd3f7274a]) successfully deleted
I20260620 18:47:57.966820 31772 ts_tablet_manager.cc:1918] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:47:57.967024 31772 log.cc:1196] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/1f00ec84dab34520a02d33fc2771535e
I20260620 18:47:57.967665 31772 ts_tablet_manager.cc:1939] T 1f00ec84dab34520a02d33fc2771535e P b004b3b50aa4474eb66cca64976a3a4b: Deleting consensus metadata
I20260620 18:47:57.969576 31256 catalog_manager.cc:4841] TS b004b3b50aa4474eb66cca64976a3a4b (127.15.102.195:42835): tablet 1f00ec84dab34520a02d33fc2771535e (table test-table [id=6d8255b2f6e94ef4a9daa51fd3f7274a]) successfully deleted
I20260620 18:47:57.970882 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:57.902616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.15.102.193:35187)
0620 18:47:57.903867 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:57.903876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:57.924828 (+ 20952us) server_negotiation.cc:407] Connection header received
0620 18:47:57.925523 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:57.925533 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:57.925652 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:57.925744 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:57.931838 (+ 6094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.933530 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.936791 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:57.937748 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:57.937896 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:57.962599 (+ 24703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:57.962612 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:57.962633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:57.962692 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:57.967134 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:57.968215 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:57.968228 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:57.968232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:57.968438 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:57.968935 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:57.968941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:57.968945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:57.969455 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:57.969886 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:57.970167 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:47:57.970453 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075}
I20260620 18:47:57.971715 31456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:58.076367 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 31333
I20260620 18:47:58.095160 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 31467
Jun 20 18:47:58 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:47:58.111467 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:58.095921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.15.102.194:36563)
0620 18:47:58.096120 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:58.096127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:58.096192 (+ 65us) server_negotiation.cc:407] Connection header received
0620 18:47:58.096418 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:58.096422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:58.096485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:58.096608 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:58.097992 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:58.098759 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:58.100363 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:58.100811 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:58.100943 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:58.105279 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:58.105295 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:58.105311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:58.105350 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:58.108308 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:58.109055 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:58.109064 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:58.109066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:58.109198 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:58.109590 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:58.109595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:58.109600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:58.110038 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:58.110348 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:58.110672 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 18:47:58.110894 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:47:58.112118 31590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:58.202785 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 31467
I20260620 18:47:58.220782 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 31601
Jun 20 18:47:58 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:47:58.236863 31747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:47:58.221590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.15.102.195:42835)
0620 18:47:58.221775 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:47:58.221780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:47:58.221799 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:47:58.221941 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:47:58.221945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:47:58.222009 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:47:58.222093 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:47:58.223437 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:58.224139 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:58.225462 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:47:58.225896 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:47:58.225990 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:47:58.231112 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:47:58.231130 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:47:58.231146 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:47:58.231192 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:47:58.234100 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:58.234902 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:58.234910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:58.234912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:58.235068 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:47:58.235481 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:47:58.235485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:47:58.235487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:47:58.235852 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:47:58.236072 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:47:58.236326 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:47:58.236494 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:47:58.237604 31724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:58.328476 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 31601
I20260620 18:47:58.345809 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 31240
I20260620 18:47:58.347007 31302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:47:58.462559 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 31240
I20260620 18:48:18.485308 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36635
--webserver_interface=127.15.102.254
--webserver_port=39073
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:48:18.824616 31780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:48:18.824990 31780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:48:18.825138 31780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:48:18.832845 31780 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:48:18.832969 31780 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:48:18.833014 31780 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:48:18.833057 31780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:48:18.833118 31780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:48:18.833189 31780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:48:18.833232 31780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:48:18.833285 31780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:48:18.843830 31780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:36635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=39073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:48:18.845228 31780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:48:18.846650 31780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:48:18.871654 31786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:18.871600 31787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:18.871650 31789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:18.872973 31780 server_base.cc:1034] running on GCE node
I20260620 18:48:18.873945 31780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:48:18.875787 31780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:48:18.877075 31780 hybrid_clock.cc:648] HybridClock initialized: now 1781981298877004 us; error 53 us; skew 500 ppm
I20260620 18:48:18.877555 31780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:48:18 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981298, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:48:18.885777 31780 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:48:18.887856 31780 webserver.cc:460] Webserver started at http://127.15.102.254:39073/ using document root <none> and password file <none>
I20260620 18:48:18.888698 31780 fs_manager.cc:359] Metadata directory not provided
I20260620 18:48:18.888816 31780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:48:18.895520 31780 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 18:48:18.898720 31796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:18.900429 31780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:48:18.900600 31780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "ef7577c548e849feae1e4761109ff3f9"
format_stamp: "Formatted at 2026-06-20 18:47:55 on dist-test-slave-zfp8"
server_key: "4d48c90a8cc16bf3c743991de4e5b91d"
server_key_iv: "536978fe22a6134aea03cb1d8288d024"
server_key_version: "encryptionkey@0"
I20260620 18:48:18.901721 31780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:48:18.939288 31780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:48:18.943856 31780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:48:18.944135 31780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:48:18.964560 31780 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:36635
I20260620 18:48:18.964592 31847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:36635 every 8 connection(s)
I20260620 18:48:18.966449 31780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:48:18.971571 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 31780
I20260620 18:48:18.972643 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:35187
--local_ip_for_outbound_sockets=127.15.102.193
--tserver_master_addrs=127.15.102.254:36635
--webserver_port=44525
--webserver_interface=127.15.102.193
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:48:18.975664 31848 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:48:18.981436 31848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Bootstrap starting.
I20260620 18:48:19.003152 31848 log.cc:824] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:48:19.017360 31848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:48:19.017976 31848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Bootstrap complete.
I20260620 18:48:19.027822 31848 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:48:19.028877 31848 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7577c548e849feae1e4761109ff3f9, State: Initialized, Role: FOLLOWER
I20260620 18:48:19.029639 31848 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:48:19.029934 31848 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:48:19.030062 31848 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:48:19.030267 31848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 18:48:19.034610 31848 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:48:19.035216 31848 leader_election.cc:302] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef7577c548e849feae1e4761109ff3f9; no voters:
I20260620 18:48:19.036008 31848 leader_election.cc:288] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 18:48:19.036161 31852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 18:48:19.037436 31852 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 2 LEADER]: Becoming Leader. State: Replica: ef7577c548e849feae1e4761109ff3f9, State: Running, Role: LEADER
I20260620 18:48:19.038144 31852 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:48:19.038558 31848 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:48:19.040966 31854 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef7577c548e849feae1e4761109ff3f9. Latest consensus state: current_term: 2 leader_uuid: "ef7577c548e849feae1e4761109ff3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } } }
I20260620 18:48:19.041007 31853 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef7577c548e849feae1e4761109ff3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } } }
I20260620 18:48:19.041384 31854 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:48:19.041464 31853 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:48:19.042335 31859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:48:19.055132 31859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:48:19.056908 31859 catalog_manager.cc:1259] Loaded cluster ID: 8eef1b28713947418e03d5d7a4291cf5
I20260620 18:48:19.057137 31859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:48:19.062871 31859 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:48:19.066468 31859 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Loaded TSK: 0
I20260620 18:48:19.109526 31859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Generated new TSK 1
I20260620 18:48:19.109951 31859 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 18:48:19.348264 31850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:48:19.348663 31850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:48:19.348810 31850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:48:19.356657 31850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:48:19.356791 31850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:48:19.357024 31850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:48:19.367684 31850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:35187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=44525
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:48:19.369701 31850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:48:19.371218 31850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:48:19.388661 31878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:19.389021 31875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:19.389501 31876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:19.390985 31850 server_base.cc:1034] running on GCE node
I20260620 18:48:19.391866 31850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:48:19.393131 31850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:48:19.394393 31850 hybrid_clock.cc:648] HybridClock initialized: now 1781981299394289 us; error 96 us; skew 500 ppm
I20260620 18:48:19.394793 31850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:48:19 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981299, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:48:19.398412 31850 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:48:19.400353 31850 webserver.cc:460] Webserver started at http://127.15.102.193:44525/ using document root <none> and password file <none>
I20260620 18:48:19.401176 31850 fs_manager.cc:359] Metadata directory not provided
I20260620 18:48:19.401338 31850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:48:19.408031 31850 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 18:48:19.411507 31885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:19.413293 31850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 18:48:19.413537 31850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "f39e9cb8e7624404ab71f828fbb64bef"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "21edfe41c9a3db5a3419028764f43a7b"
server_key_iv: "3d04152e23d2e92f969e4472a13cecdd"
server_key_version: "encryptionkey@0"
I20260620 18:48:19.414842 31850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:48:19.444363 31850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:48:19.448815 31850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:48:19.449132 31850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:48:19.450275 31850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:48:19.451890 31850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:48:19.452021 31850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:19.452173 31850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:48:19.452265 31850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:19.487535 31850 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:35187
I20260620 18:48:19.487623 31997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:35187 every 8 connection(s)
I20260620 18:48:19.489099 31850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:48:19.491847 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 31850
I20260620 18:48:19.492763 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:36563
--local_ip_for_outbound_sockets=127.15.102.194
--tserver_master_addrs=127.15.102.254:36635
--webserver_port=35451
--webserver_interface=127.15.102.194
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:48:19 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981299, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:48:19.518833 32001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:19.492593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:57387)
0620 18:48:19.493596 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:19.493651 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 18:48:19.496589 (+ 2938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:48:19.498230 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:48:19.498244 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:48:19.499061 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:48:19.499973 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:19.500002 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:19.501819 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:19.501827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:19.503460 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:19.503475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:19.505237 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:19.505247 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:19.505515 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:19.506282 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:48:19.506306 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:48:19.508704 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:48:19.513933 (+ 5229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:48:19.513941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:48:19.513957 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:48:19.514532 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:48:19.515150 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:48:19.515153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:48:19.515155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:48:19.515406 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:48:19.516546 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:48:19.516552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:48:19.516910 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 18:48:19.517159 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 18:48:19.517607 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":389,"threads_started":1}
I20260620 18:48:19.519490 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:19.492362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:57387 (local address 127.15.102.254:36635)
0620 18:48:19.493492 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:19.493508 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:19.493808 (+ 300us) server_negotiation.cc:407] Connection header received
0620 18:48:19.497222 (+ 3414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:19.497237 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:19.497643 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:19.497971 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:19.500256 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:19.501578 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:19.503646 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:19.504946 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:19.505174 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:19.509091 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:19.509105 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:19.509132 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:19.509193 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:19.513711 (+ 4518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:19.514723 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:19.514736 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:19.514747 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:19.515010 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:19.515542 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:19.515550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:19.515556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:19.516393 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:19.516643 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:19.517486 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 18:48:19.518031 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":342,"threads_started":1}
I20260620 18:48:19.522552 31998 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36635
I20260620 18:48:19.522902 31998 heartbeater.cc:464] Registering TS with master...
I20260620 18:48:19.523730 31998 heartbeater.cc:511] Master 127.15.102.254:36635 requested a full tablet report, sending...
I20260620 18:48:19.526117 31813 ts_manager.cc:194] Registered new tserver with Master: f39e9cb8e7624404ab71f828fbb64bef (127.15.102.193:35187)
I20260620 18:48:19.527694 31813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:57387
W20260620 18:48:19.862191 32002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:48:19.862556 32002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:48:19.862733 32002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:48:19.870638 32002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:48:19.870828 32002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:48:19.871006 32002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:48:19.881371 32002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:36563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=35451
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:48:19.882810 32002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:48:19.884313 32002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:48:19.901105 32008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:19.901105 32007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:19.902208 32010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:19.902551 32002 server_base.cc:1034] running on GCE node
I20260620 18:48:19.903287 32002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:48:19.904531 32002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:48:19.905784 32002 hybrid_clock.cc:648] HybridClock initialized: now 1781981299905708 us; error 65 us; skew 500 ppm
I20260620 18:48:19.906153 32002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:48:19 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981299, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:48:19.909927 32002 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:48:19.911818 32002 webserver.cc:460] Webserver started at http://127.15.102.194:35451/ using document root <none> and password file <none>
I20260620 18:48:19.912652 32002 fs_manager.cc:359] Metadata directory not provided
I20260620 18:48:19.912827 32002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:48:19.919602 32002 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:48:19.923081 32017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:19.924762 32002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:48:19.925000 32002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "afe52d9462fd41adb277761ade3a162b"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "af934a0413ec5da09c0ec61b3754ccb5"
server_key_iv: "ae2bf23c00f2d0309e017f58fd6c1194"
server_key_version: "encryptionkey@0"
I20260620 18:48:19.926128 32002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:48:19.946276 32002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:48:19.951079 32002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:48:19.951435 32002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:48:19.952490 32002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:48:19.953946 32002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:48:19.954084 32002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:19.954229 32002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:48:19.954332 32002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:19.989743 32002 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:36563
I20260620 18:48:19.989794 32129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:36563 every 8 connection(s)
I20260620 18:48:19.991704 32002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:48:19.994396 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 32002
I20260620 18:48:19.995225 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:42835
--local_ip_for_outbound_sockets=127.15.102.195
--tserver_master_addrs=127.15.102.254:36635
--webserver_port=36305
--webserver_interface=127.15.102.195
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 18:48:20 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981299, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:48:20.017880 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:19.994918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:43621 (local address 127.15.102.254:36635)
0620 18:48:19.995158 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:19.995165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:19.996845 (+ 1680us) server_negotiation.cc:407] Connection header received
0620 18:48:20.000048 (+ 3203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.000053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.000157 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.000268 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:20.002849 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.003647 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.005521 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.006082 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.006178 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.010815 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:20.010821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:20.010825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:20.010868 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:20.013823 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.014958 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.014967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.014971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.015142 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.015778 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.015784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.015790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.016130 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:20.016320 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.017172 (+ 852us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.017452 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:48:20.019291 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:19.995651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:43621)
0620 18:48:19.996616 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:19.996677 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 18:48:19.999660 (+ 2983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:48:20.000624 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:48:20.000641 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:48:20.001629 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:48:20.002612 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.002645 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.003856 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.003865 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.005266 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.005280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.006553 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.006563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.006891 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.007706 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:48:20.007732 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:48:20.010483 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:48:20.014059 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:48:20.014071 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:48:20.014097 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:48:20.014725 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:48:20.015307 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:48:20.015315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:48:20.015319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:48:20.015644 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:48:20.016293 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:48:20.016303 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:48:20.016738 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 18:48:20.017070 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 18:48:20.017679 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":335,"threads_started":1}
I20260620 18:48:20.021294 32130 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36635
I20260620 18:48:20.021706 32130 heartbeater.cc:464] Registering TS with master...
I20260620 18:48:20.022792 32130 heartbeater.cc:511] Master 127.15.102.254:36635 requested a full tablet report, sending...
I20260620 18:48:20.024654 31813 ts_manager.cc:194] Registered new tserver with Master: afe52d9462fd41adb277761ade3a162b (127.15.102.194:36563)
I20260620 18:48:20.025661 31813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:43621
W20260620 18:48:20.349282 32133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:48:20.349614 32133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:48:20.349781 32133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:48:20.357477 32133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:48:20.357612 32133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:48:20.357864 32133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:48:20.368095 32133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:42835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=36305
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:48:20.369585 32133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:48:20.371012 32133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:48:20.385232 32138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:20.385246 32141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:20.385270 32139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:20.386106 32133 server_base.cc:1034] running on GCE node
I20260620 18:48:20.386806 32133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:48:20.388007 32133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:48:20.389289 32133 hybrid_clock.cc:648] HybridClock initialized: now 1781981300389237 us; error 46 us; skew 500 ppm
I20260620 18:48:20.389721 32133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:48:20 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981300, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:48:20.393392 32133 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:48:20.395092 32133 webserver.cc:460] Webserver started at http://127.15.102.195:36305/ using document root <none> and password file <none>
I20260620 18:48:20.395897 32133 fs_manager.cc:359] Metadata directory not provided
I20260620 18:48:20.396034 32133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:48:20.402077 32133 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 18:48:20.405225 32148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:20.406994 32133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:48:20.407166 32133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "b004b3b50aa4474eb66cca64976a3a4b"
format_stamp: "Formatted at 2026-06-20 18:47:57 on dist-test-slave-zfp8"
server_key: "8119525566168b1ea84b47668f347b43"
server_key_iv: "422280abb9f1912e5b9b54b0179b9210"
server_key_version: "encryptionkey@0"
I20260620 18:48:20.408330 32133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:48:20.431083 32133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:48:20.436311 32133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:48:20.436573 32133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:48:20.437638 32133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:48:20.439185 32133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:48:20.439361 32133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 18:48:20.439502 32133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:48:20.439587 32133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:20.479786 32133 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:42835
I20260620 18:48:20.479831 32260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:42835 every 8 connection(s)
I20260620 18:48:20.481370 32133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:48:20.484663 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 32133
Jun 20 18:48:20 dist-test-slave-zfp8 krb5kdc[31227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981300, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:48:20.511765 32000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.484420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:56929 (local address 127.15.102.254:36635)
0620 18:48:20.484679 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.484686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.486196 (+ 1510us) server_negotiation.cc:407] Connection header received
0620 18:48:20.490033 (+ 3837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.490044 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.490188 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.490284 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:20.493521 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.495072 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.498572 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.499779 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.499995 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.504256 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:20.504261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:20.504265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:20.504310 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:20.507948 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.508961 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.508968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.508972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.509153 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.509626 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.509632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.509636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.510073 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:20.510271 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.511035 (+ 764us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.511392 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 18:48:20.513051 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.484961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:56929)
0620 18:48:20.485945 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:20.486025 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 18:48:20.489555 (+ 3530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:48:20.490999 (+ 1444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:48:20.491018 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:48:20.492078 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:48:20.493198 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.493236 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.495433 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.495442 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.498277 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.498302 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.500068 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.500074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.500326 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.501139 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:48:20.501172 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:48:20.503974 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:48:20.508124 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:48:20.508140 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:48:20.508156 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:48:20.508794 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:48:20.509286 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:48:20.509290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:48:20.509293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:48:20.509488 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:48:20.510219 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:48:20.510226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:48:20.510602 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 18:48:20.510854 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 18:48:20.511418 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":435,"threads_started":1}
I20260620 18:48:20.515005 32261 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36635
I20260620 18:48:20.515446 32261 heartbeater.cc:464] Registering TS with master...
I20260620 18:48:20.516294 32261 heartbeater.cc:511] Master 127.15.102.254:36635 requested a full tablet report, sending...
I20260620 18:48:20.518220 31813 ts_manager.cc:194] Registered new tserver with Master: b004b3b50aa4474eb66cca64976a3a4b (127.15.102.195:42835)
I20260620 18:48:20.518355 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.485683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.15.102.254:36635)
0620 18:48:20.499746 (+ 14063us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.499750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.501244 (+ 1494us) server_negotiation.cc:407] Connection header received
0620 18:48:20.501427 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.501435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.501534 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.501662 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:20.503562 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.504591 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.506916 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.507484 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.507586 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.511460 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:20.511466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:20.511471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:20.511507 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:20.515085 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.515917 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.515922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.515924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.516046 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.516478 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.516482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.516484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.516864 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:20.517149 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.517633 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.517878 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13956,"thread_start_us":13825,"threads_started":1}
I20260620 18:48:20.519342 31813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:56929
I20260620 18:48:20.522009 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:48:20.529099 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.523014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.15.102.254:36635)
0620 18:48:20.523268 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.523278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.523692 (+ 414us) server_negotiation.cc:407] Connection header received
0620 18:48:20.523874 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.523878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.524022 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.524182 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:48:20.524598 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.525325 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.526996 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.527558 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.527660 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.528198 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:48:20.528430 (+ 232us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 18:48:20.528743 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35844: token expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 18:48:20.529260 32264 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35844: token expired
W20260620 18:48:20.529857 31736 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.102.254:36635)
I20260620 18:48:20.531133 31998 heartbeater.cc:503] Master 127.15.102.254:36635 was elected leader, sending a full tablet report...
I20260620 18:48:20.543311 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.530376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.15.102.254:36635)
0620 18:48:20.530545 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.530549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.530564 (+ 15us) server_negotiation.cc:407] Connection header received
0620 18:48:20.530681 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.530684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.530741 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.530819 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:20.532570 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.533442 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.534774 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.535149 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.535299 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.537921 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:20.537926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:20.537929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:20.537958 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:20.540766 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.541487 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.541490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.541493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.541625 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.541958 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.541961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.541963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.542296 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:20.542486 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.542796 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.542974 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:48:20.547115 31812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:48:20.555049 31812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:48:20.592854 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.568982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.15.102.194:36563)
0620 18:48:20.580292 (+ 11310us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.580306 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.582526 (+ 2220us) server_negotiation.cc:407] Connection header received
0620 18:48:20.582803 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.582813 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.583076 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.583379 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:48:20.584529 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.586153 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.588938 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.590691 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.590911 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.591141 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.592336 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.592522 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11020,"thread_start_us":10857,"threads_started":1}
I20260620 18:48:20.592893 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.568687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:36563 (local address 127.0.0.1:47146)
0620 18:48:20.582328 (+ 13641us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:20.582354 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:48:20.582564 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:48:20.583787 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:48:20.583792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:48:20.583822 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:48:20.584301 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.584316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.586345 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.586351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.588725 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.588736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.591452 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.591460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.591797 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.591997 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 18:48:20.592132 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 18:48:20.592409 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13493,"mutex_wait_us":140,"thread_start_us":13303,"threads_started":1}
I20260620 18:48:20.593019 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.568545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.15.102.193:35187)
0620 18:48:20.582088 (+ 13543us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.582101 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.582141 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:48:20.582271 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.582279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.582596 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.582852 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:48:20.583750 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.585527 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.588954 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.590691 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.590936 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.591141 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.592308 (+ 1167us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.592538 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13296,"thread_start_us":13110,"threads_started":1}
I20260620 18:48:20.594137 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.568240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:35187 (local address 127.0.0.1:42412)
0620 18:48:20.581122 (+ 12882us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:20.581195 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 18:48:20.581536 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:48:20.582875 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:48:20.582883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:48:20.582919 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:48:20.583526 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.583551 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.585745 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.585751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.588687 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.588711 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.591452 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.591460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.591798 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.591970 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 18:48:20.592133 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 18:48:20.593492 (+ 1359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12424,"mutex_wait_us":99,"thread_start_us":12185,"threads_started":1}
I20260620 18:48:20.597715 32065 tablet_service.cc:1467] Processing CreateTablet for tablet c3f76bcbb37a4846b4a464397398cb7c (DEFAULT_TABLE table=test-table [id=5aeccc82b8c24af798cfaa3baa220040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:48:20.598654 31933 tablet_service.cc:1467] Processing CreateTablet for tablet c3f76bcbb37a4846b4a464397398cb7c (DEFAULT_TABLE table=test-table [id=5aeccc82b8c24af798cfaa3baa220040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:48:20.599577 32065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f76bcbb37a4846b4a464397398cb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:48:20.600404 31933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f76bcbb37a4846b4a464397398cb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:48:20.600857 32275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.583505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:42835 (local address 127.0.0.1:52144)
0620 18:48:20.590300 (+ 6795us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:20.590332 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 18:48:20.590531 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:48:20.593500 (+ 2969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:48:20.593506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:48:20.593540 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:48:20.594012 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.594028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.596588 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.596595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.598281 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.598297 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.599934 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.599940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.600124 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.600170 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:48:20.600292 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 18:48:20.600449 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6575,"thread_start_us":7137,"threads_started":1}
I20260620 18:48:20.600873 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.583584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.15.102.195:42835)
0620 18:48:20.592336 (+ 8752us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.592347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.592389 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:48:20.592515 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.592524 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.592851 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.593138 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:48:20.594290 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.596423 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.598501 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.599692 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.599918 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.600104 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.600364 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.600525 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8552,"thread_start_us":8357,"threads_started":1}
I20260620 18:48:20.604964 32196 tablet_service.cc:1467] Processing CreateTablet for tablet c3f76bcbb37a4846b4a464397398cb7c (DEFAULT_TABLE table=test-table [id=5aeccc82b8c24af798cfaa3baa220040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:48:20.606453 32196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f76bcbb37a4846b4a464397398cb7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:48:20.617645 32278 tablet_bootstrap.cc:492] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Bootstrap starting.
I20260620 18:48:20.619047 32277 tablet_bootstrap.cc:492] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Bootstrap starting.
I20260620 18:48:20.621577 32279 tablet_bootstrap.cc:492] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Bootstrap starting.
I20260620 18:48:20.623512 32277 tablet_bootstrap.cc:654] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Neither blocks nor log segments found. Creating new log.
I20260620 18:48:20.623803 32278 tablet_bootstrap.cc:654] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Neither blocks nor log segments found. Creating new log.
I20260620 18:48:20.624707 32277 log.cc:824] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Log is configured to *not* fsync() on all Append() calls
I20260620 18:48:20.625181 32278 log.cc:824] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:48:20.625547 32279 tablet_bootstrap.cc:654] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Neither blocks nor log segments found. Creating new log.
I20260620 18:48:20.626911 32279 log.cc:824] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:48:20.628597 32277 tablet_bootstrap.cc:492] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: No bootstrap required, opened a new log
I20260620 18:48:20.628643 32278 tablet_bootstrap.cc:492] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: No bootstrap required, opened a new log
I20260620 18:48:20.628952 32278 ts_tablet_manager.cc:1397] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260620 18:48:20.628966 32277 ts_tablet_manager.cc:1397] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 18:48:20.630842 32279 tablet_bootstrap.cc:492] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: No bootstrap required, opened a new log
I20260620 18:48:20.631173 32279 ts_tablet_manager.cc:1397] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260620 18:48:20.637104 32278 raft_consensus.cc:348] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.637584 32278 raft_consensus.cc:374] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:48:20.637744 32278 raft_consensus.cc:729] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe52d9462fd41adb277761ade3a162b, State: Initialized, Role: FOLLOWER
I20260620 18:48:20.638530 32278 consensus_queue.cc:260] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.640087 32130 heartbeater.cc:503] Master 127.15.102.254:36635 was elected leader, sending a full tablet report...
I20260620 18:48:20.640831 32278 ts_tablet_manager.cc:1428] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260620 18:48:20.642905 32279 raft_consensus.cc:348] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.641676 32277 raft_consensus.cc:348] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.643508 32279 raft_consensus.cc:374] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:48:20.643602 32277 raft_consensus.cc:374] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:48:20.643664 32279 raft_consensus.cc:729] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b004b3b50aa4474eb66cca64976a3a4b, State: Initialized, Role: FOLLOWER
I20260620 18:48:20.643744 32277 raft_consensus.cc:729] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39e9cb8e7624404ab71f828fbb64bef, State: Initialized, Role: FOLLOWER
I20260620 18:48:20.644404 32277 consensus_queue.cc:260] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.644512 32279 consensus_queue.cc:260] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.646425 32277 ts_tablet_manager.cc:1428] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Time spent starting tablet: real 0.017s user 0.008s sys 0.006s
I20260620 18:48:20.650080 32261 heartbeater.cc:503] Master 127.15.102.254:36635 was elected leader, sending a full tablet report...
I20260620 18:48:20.650980 32279 ts_tablet_manager.cc:1428] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Time spent starting tablet: real 0.020s user 0.011s sys 0.007s
I20260620 18:48:20.694047 32283 raft_consensus.cc:482] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:48:20.694380 32283 raft_consensus.cc:504] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.696275 32283 leader_election.cc:288] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f39e9cb8e7624404ab71f828fbb64bef (127.15.102.193:35187), b004b3b50aa4474eb66cca64976a3a4b (127.15.102.195:42835)
I20260620 18:48:20.703812 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.696731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:37063 (local address 127.15.102.193:35187)
0620 18:48:20.696961 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.696970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.697917 (+ 947us) server_negotiation.cc:407] Connection header received
0620 18:48:20.698095 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.698100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.698181 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.698279 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:48:20.698900 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.699791 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.701806 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.702674 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.702813 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.702870 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.703381 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.703539 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:48:20.703924 32287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.696549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:35187 (local address 127.15.102.194:37063)
0620 18:48:20.697785 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:20.697807 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:48:20.697939 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:48:20.698317 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:48:20.698321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:48:20.698346 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:48:20.698740 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.698751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.699926 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.699931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.701615 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.701635 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.702909 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.702915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.703098 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.703163 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 18:48:20.703336 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 18:48:20.703453 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":37,"thread_start_us":914,"threads_started":1}
I20260620 18:48:20.703956 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.696997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:47741 (local address 127.15.102.195:42835)
0620 18:48:20.697189 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.697196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.698107 (+ 911us) server_negotiation.cc:407] Connection header received
0620 18:48:20.698297 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.698303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.698388 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.698489 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:48:20.699169 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.700243 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.702039 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.702909 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.703068 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.703156 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.703566 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.703713 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 18:48:20.704054 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.696728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:42835 (local address 127.15.102.194:47741)
0620 18:48:20.697976 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:20.697995 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:48:20.698139 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:48:20.698524 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:48:20.698527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:48:20.698547 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:48:20.698987 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.699003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.700374 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.700378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.701881 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:48:20.701895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.703108 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.703112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:48:20.703391 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.703424 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 18:48:20.703518 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 18:48:20.703644 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"mutex_wait_us":25,"thread_start_us":555,"threads_started":1}
I20260620 18:48:20.705297 31953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f76bcbb37a4846b4a464397398cb7c" candidate_uuid: "afe52d9462fd41adb277761ade3a162b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39e9cb8e7624404ab71f828fbb64bef" is_pre_election: true
I20260620 18:48:20.705333 32216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f76bcbb37a4846b4a464397398cb7c" candidate_uuid: "afe52d9462fd41adb277761ade3a162b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b004b3b50aa4474eb66cca64976a3a4b" is_pre_election: true
I20260620 18:48:20.705854 31953 raft_consensus.cc:2454] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe52d9462fd41adb277761ade3a162b in term 0.
I20260620 18:48:20.705889 32216 raft_consensus.cc:2454] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afe52d9462fd41adb277761ade3a162b in term 0.
I20260620 18:48:20.706629 32020 leader_election.cc:302] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe52d9462fd41adb277761ade3a162b, f39e9cb8e7624404ab71f828fbb64bef; no voters:
I20260620 18:48:20.707041 32283 raft_consensus.cc:2781] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:48:20.707201 32283 raft_consensus.cc:482] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:48:20.707360 32283 raft_consensus.cc:3037] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:48:20.709563 32283 raft_consensus.cc:504] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.710285 32283 leader_election.cc:288] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [CANDIDATE]: Term 1 election: Requested vote from peers f39e9cb8e7624404ab71f828fbb64bef (127.15.102.193:35187), b004b3b50aa4474eb66cca64976a3a4b (127.15.102.195:42835)
I20260620 18:48:20.710747 31953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f76bcbb37a4846b4a464397398cb7c" candidate_uuid: "afe52d9462fd41adb277761ade3a162b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39e9cb8e7624404ab71f828fbb64bef"
I20260620 18:48:20.710963 31953 raft_consensus.cc:3037] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:48:20.710899 32216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f76bcbb37a4846b4a464397398cb7c" candidate_uuid: "afe52d9462fd41adb277761ade3a162b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b004b3b50aa4474eb66cca64976a3a4b"
I20260620 18:48:20.711189 32216 raft_consensus.cc:3037] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:48:20.713496 31953 raft_consensus.cc:2454] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe52d9462fd41adb277761ade3a162b in term 1.
I20260620 18:48:20.713968 32020 leader_election.cc:302] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afe52d9462fd41adb277761ade3a162b, f39e9cb8e7624404ab71f828fbb64bef; no voters:
I20260620 18:48:20.714294 32283 raft_consensus.cc:2781] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:48:20.714599 32216 raft_consensus.cc:2454] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afe52d9462fd41adb277761ade3a162b in term 1.
I20260620 18:48:20.714851 32283 raft_consensus.cc:686] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 1 LEADER]: Becoming Leader. State: Replica: afe52d9462fd41adb277761ade3a162b, State: Running, Role: LEADER
I20260620 18:48:20.715595 32283 consensus_queue.cc:237] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } }
I20260620 18:48:20.729108 31813 catalog_manager.cc:5495] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b reported cstate change: term changed from 0 to 1, leader changed from <none> to afe52d9462fd41adb277761ade3a162b (127.15.102.194). New cstate: current_term: 1 leader_uuid: "afe52d9462fd41adb277761ade3a162b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afe52d9462fd41adb277761ade3a162b" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 36563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 } health_report { overall_health: UNKNOWN } } }
W20260620 18:48:20.733135 32262 tablet.cc:2369] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:48:20.741566 31999 tablet.cc:2369] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:48:20.744239 32131 tablet.cc:2369] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:48:20.767879 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.761902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.15.102.194:36563)
0620 18:48:20.762171 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.762177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.762197 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:48:20.762407 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.762411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.762493 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.762639 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:48:20.763019 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.763853 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.765541 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.765974 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.766096 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.766621 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:48:20.767150 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:48:20.767361 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.767454 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.767545 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 18:48:20.776752 31953 raft_consensus.cc:1261] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 1 FOLLOWER]: Refusing update from remote peer afe52d9462fd41adb277761ade3a162b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:48:20.776753 32216 raft_consensus.cc:1261] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 1 FOLLOWER]: Refusing update from remote peer afe52d9462fd41adb277761ade3a162b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:48:20.777793 32288 consensus_queue.cc:1035] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b004b3b50aa4474eb66cca64976a3a4b" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:48:20.778203 32283 consensus_queue.cc:1035] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39e9cb8e7624404ab71f828fbb64bef" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:48:20.792207 32293 mvcc.cc:205] Tried to move back new op lower bound from 7298995407970488320 to 7298995407738007552. Current Snapshot: MvccSnapshot[applied={T|T < 7298995407970488320}]
I20260620 18:48:20.795217 32292 mvcc.cc:205] Tried to move back new op lower bound from 7298995407970488320 to 7298995407738007552. Current Snapshot: MvccSnapshot[applied={T|T < 7298995407970488320}]
I20260620 18:48:20.810290 31812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:48:20.810621 31812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:48:20.814879 31812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Sending DeleteTablet for 3 replicas of tablet c3f76bcbb37a4846b4a464397398cb7c
I20260620 18:48:20.816315 31933 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f76bcbb37a4846b4a464397398cb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:48:20 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:42412
I20260620 18:48:20.816491 32196 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f76bcbb37a4846b4a464397398cb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:48:20 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:52144
I20260620 18:48:20.816713 32065 tablet_service.cc:1514] Processing DeleteTablet for tablet c3f76bcbb37a4846b4a464397398cb7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:48:20 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:47146
I20260620 18:48:20.818228 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 31850
I20260620 18:48:20.827476 32303 tablet_replica.cc:330] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: stopping tablet replica
I20260620 18:48:20.827457 32302 tablet_replica.cc:330] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: stopping tablet replica
I20260620 18:48:20.827678 32301 tablet_replica.cc:330] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: stopping tablet replica
I20260620 18:48:20.828068 32303 raft_consensus.cc:2229] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:48:20.828076 32301 raft_consensus.cc:2229] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:48:20.828588 32302 raft_consensus.cc:2229] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:48:20.828703 32301 raft_consensus.cc:2258] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:48:20.828768 32303 raft_consensus.cc:2258] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:48:20.829355 32302 raft_consensus.cc:2258] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:48:20.831286 32301 ts_tablet_manager.cc:1905] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:48:20.831277 32303 ts_tablet_manager.cc:1905] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:48:20.832005 32302 ts_tablet_manager.cc:1905] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:48:20.838828 32303 ts_tablet_manager.cc:1918] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:48:20.838868 32301 ts_tablet_manager.cc:1918] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:48:20.839061 32301 log.cc:1196] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/c3f76bcbb37a4846b4a464397398cb7c
I20260620 18:48:20.839071 32303 log.cc:1196] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/c3f76bcbb37a4846b4a464397398cb7c
I20260620 18:48:20.839597 32302 ts_tablet_manager.cc:1918] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:48:20.839679 32301 ts_tablet_manager.cc:1939] T c3f76bcbb37a4846b4a464397398cb7c P f39e9cb8e7624404ab71f828fbb64bef: Deleting consensus metadata
I20260620 18:48:20.839689 32303 ts_tablet_manager.cc:1939] T c3f76bcbb37a4846b4a464397398cb7c P afe52d9462fd41adb277761ade3a162b: Deleting consensus metadata
I20260620 18:48:20.839867 32302 log.cc:1196] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/c3f76bcbb37a4846b4a464397398cb7c
I20260620 18:48:20.840462 32302 ts_tablet_manager.cc:1939] T c3f76bcbb37a4846b4a464397398cb7c P b004b3b50aa4474eb66cca64976a3a4b: Deleting consensus metadata
I20260620 18:48:20.841673 31799 catalog_manager.cc:4841] TS f39e9cb8e7624404ab71f828fbb64bef (127.15.102.193:35187): tablet c3f76bcbb37a4846b4a464397398cb7c (table test-table [id=5aeccc82b8c24af798cfaa3baa220040]) successfully deleted
I20260620 18:48:20.842200 31797 catalog_manager.cc:4841] TS afe52d9462fd41adb277761ade3a162b (127.15.102.194:36563): tablet c3f76bcbb37a4846b4a464397398cb7c (table test-table [id=5aeccc82b8c24af798cfaa3baa220040]) successfully deleted
I20260620 18:48:20.842461 31797 catalog_manager.cc:4841] TS b004b3b50aa4474eb66cca64976a3a4b (127.15.102.195:42835): tablet c3f76bcbb37a4846b4a464397398cb7c (table test-table [id=5aeccc82b8c24af798cfaa3baa220040]) successfully deleted
I20260620 18:48:20.844895 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.820392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.15.102.193:35187)
0620 18:48:20.820627 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.820637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.820662 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:48:20.820745 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.820753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.820895 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.821038 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:20.823915 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.825611 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.828016 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.828765 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.828913 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.833090 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:20.833111 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:20.833129 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:20.833189 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:20.837549 (+ 4360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.842383 (+ 4834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.842394 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.842398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.842606 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.843092 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.843098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.843101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.843611 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:20.843929 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.844242 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.844494 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:48:20.845642 31993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:48:20.952494 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 31850
I20260620 18:48:20.971482 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 32002
I20260620 18:48:20.992239 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:20.972405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.15.102.194:36563)
0620 18:48:20.972595 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:20.972603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:20.972660 (+ 57us) server_negotiation.cc:407] Connection header received
0620 18:48:20.972904 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:20.972908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:20.972989 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:20.973100 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:20.974511 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.975546 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.976847 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:20.977345 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:20.977494 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:20.984574 (+ 7080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:20.984590 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:20.984608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:20.984652 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:20.988006 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.988844 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.988853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.988855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.989014 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:20.989420 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:20.989432 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:20.989438 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:20.989915 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:20.990282 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:20.991567 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0620 18:48:20.991834 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 18:48:20.993078 32125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:48:21.111428 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 32002
I20260620 18:48:21.131445 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 32133
I20260620 18:48:21.144729 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:21.132266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.15.102.195:42835)
0620 18:48:21.132440 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:21.132445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:21.132539 (+ 94us) server_negotiation.cc:407] Connection header received
0620 18:48:21.132716 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:21.132720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:21.132788 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:21.132874 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:21.134242 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:21.134976 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:21.136297 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:21.136692 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:21.136796 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:21.139489 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:21.139503 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:21.139520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:21.139553 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:21.142158 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:21.142782 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:21.142789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:21.142792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:21.142915 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:21.143371 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:21.143378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:21.143384 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:21.143777 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:21.144001 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:21.144296 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 18:48:21.144463 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:48:21.145364 32256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:48:21.235193 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 32133
I20260620 18:48:21.252835 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 31780
I20260620 18:48:21.254112 31843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:48:21.371512 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 31780
I20260620 18:48:41.393996 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36635
--webserver_interface=127.15.102.254
--webserver_port=39073
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:48:41.727373 32308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:48:41.727741 32308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:48:41.727900 32308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:48:41.735107 32308 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 18:48:41.735276 32308 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 18:48:41.735373 32308 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 18:48:41.735469 32308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:48:41.735553 32308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:48:41.735664 32308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:48:41.735747 32308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:48:41.735826 32308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:48:41.746552 32308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:36635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=39073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:48:41.748019 32308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:48:41.749409 32308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:48:41.761862 32313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:41.761865 32314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:41.762113 32308 server_base.cc:1034] running on GCE node
W20260620 18:48:41.761804 32316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:41.763137 32308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:48:41.764699 32308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:48:41.765947 32308 hybrid_clock.cc:648] HybridClock initialized: now 1781981321765883 us; error 32 us; skew 500 ppm
I20260620 18:48:41.766317 32308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:48:41 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:48:41.772779 32308 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:48:41.774276 32308 webserver.cc:460] Webserver started at http://127.15.102.254:39073/ using document root <none> and password file <none>
I20260620 18:48:41.775008 32308 fs_manager.cc:359] Metadata directory not provided
I20260620 18:48:41.775146 32308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:48:41.780606 32308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 18:48:41.783511 32323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:41.784961 32308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:48:41.785166 32308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "ef7577c548e849feae1e4761109ff3f9"
format_stamp: "Formatted at 2026-06-20 18:47:55 on dist-test-slave-zfp8"
server_key: "4d48c90a8cc16bf3c743991de4e5b91d"
server_key_iv: "536978fe22a6134aea03cb1d8288d024"
server_key_version: "encryptionkey@0"
I20260620 18:48:41.786151 32308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:48:41.808761 32308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:48:41.815043 32308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:48:41.815402 32308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:48:41.837879 32308 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:36635
I20260620 18:48:41.837942 32374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:36635 every 8 connection(s)
I20260620 18:48:41.840000 32308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:48:41.848495 32375 sys_catalog.cc:263] Verifying existing consensus state
I20260620 18:48:41.848620 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 32308
I20260620 18:48:41.849414 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:35187
--local_ip_for_outbound_sockets=127.15.102.193
--tserver_master_addrs=127.15.102.254:36635
--webserver_port=44525
--webserver_interface=127.15.102.193
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:48:41.852035 32375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Bootstrap starting.
I20260620 18:48:41.877976 32375 log.cc:824] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:48:41.895507 32375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 18:48:41.896073 32375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Bootstrap complete.
I20260620 18:48:41.905320 32375 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:48:41.906327 32375 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7577c548e849feae1e4761109ff3f9, State: Initialized, Role: FOLLOWER
I20260620 18:48:41.907051 32375 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:48:41.907344 32375 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:48:41.907493 32375 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:48:41.907680 32375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 18:48:41.911746 32375 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:48:41.912278 32375 leader_election.cc:302] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef7577c548e849feae1e4761109ff3f9; no voters:
I20260620 18:48:41.913023 32375 leader_election.cc:288] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 18:48:41.913177 32379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 18:48:41.914424 32379 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [term 3 LEADER]: Becoming Leader. State: Replica: ef7577c548e849feae1e4761109ff3f9, State: Running, Role: LEADER
I20260620 18:48:41.915107 32379 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } }
I20260620 18:48:41.915524 32375 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:48:41.917846 32381 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef7577c548e849feae1e4761109ff3f9. Latest consensus state: current_term: 3 leader_uuid: "ef7577c548e849feae1e4761109ff3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } } }
I20260620 18:48:41.918294 32381 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:48:41.918833 32380 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ef7577c548e849feae1e4761109ff3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7577c548e849feae1e4761109ff3f9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36635 } } }
I20260620 18:48:41.919165 32380 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:48:41.921763 32385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:48:41.935021 32385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:48:41.936635 32385 catalog_manager.cc:1259] Loaded cluster ID: 8eef1b28713947418e03d5d7a4291cf5
I20260620 18:48:41.936990 32385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:48:41.942024 32385 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:48:41.943800 32385 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Loaded TSK: 0
I20260620 18:48:41.944034 32385 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef7577c548e849feae1e4761109ff3f9: Loaded TSK: 1
F20260620 18:48:41.945207 32385 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faa6968f9ad google::LogMessage::Fail() at ??:0
@ 0x7faa6969288a google::LogMessage::SendToLog() at ??:0
@ 0x7faa6968f4de google::LogMessage::Flush() at ??:0
@ 0x7faa69690a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faa6d3f37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faa6d3e273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faa7f0be1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faa7f0bd783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faa7f1492a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7faa7f1490c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7faa7f1935e9 std::function<>::operator()() at ??:0
@ 0x7faa7f0c1cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7faa7f0bfc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faa7f147610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7faa7f147439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7faa6c2db345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faa6c2de91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7faa6c2de799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7faa6c2b9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7faa6f5936db start_thread at ??:0
@ 0x7faa6852961f clone at ??:0
W20260620 18:48:42.241737 32377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:48:42.242076 32377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:48:42.242240 32377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:48:42.249686 32377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:48:42.249845 32377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:48:42.250023 32377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:48:42.260121 32377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:35187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=44525
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:48:42.261518 32377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:48:42.262874 32377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:48:42.277102 32425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:42.277156 32422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:42.277182 32423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:42.278136 32377 server_base.cc:1034] running on GCE node
I20260620 18:48:42.278784 32377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:48:42.279855 32377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:48:42.281086 32377 hybrid_clock.cc:648] HybridClock initialized: now 1781981322281033 us; error 48 us; skew 500 ppm
I20260620 18:48:42.281409 32377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:48:42 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:48:42.284791 32377 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:48:42.286291 32377 webserver.cc:460] Webserver started at http://127.15.102.193:44525/ using document root <none> and password file <none>
I20260620 18:48:42.286993 32377 fs_manager.cc:359] Metadata directory not provided
I20260620 18:48:42.287142 32377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:48:42.292470 32377 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:48:42.295343 32432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:42.296696 32377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:48:42.296834 32377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "f39e9cb8e7624404ab71f828fbb64bef"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "21edfe41c9a3db5a3419028764f43a7b"
server_key_iv: "3d04152e23d2e92f969e4472a13cecdd"
server_key_version: "encryptionkey@0"
I20260620 18:48:42.297820 32377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:48:42.309650 32377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:48:42.313495 32377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:48:42.313726 32377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:48:42.314873 32377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:48:42.316267 32377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:48:42.316365 32377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:42.316442 32377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:48:42.316498 32377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:42.349609 32377 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:35187
I20260620 18:48:42.349704 32544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:35187 every 8 connection(s)
I20260620 18:48:42.351110 32377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:48:42.356058 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:42.354166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:44905)
0620 18:48:42.354892 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:42.355132 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":281,"threads_started":1}
W20260620 18:48:42.356621 32545 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:36635 (0 consecutive failures): Network error: Failed to ping master at 127.15.102.254:36635: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
I20260620 18:48:42.357416 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:42.357111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:56131)
0620 18:48:42.357244 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:42.357311 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 18:48:42.358230 32547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:42.357939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:37771)
0620 18:48:42.358053 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:42.358129 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:48:42.358515 32545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:48:42.359001 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 32377
I20260620 18:48:42.359653 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:36563
--local_ip_for_outbound_sockets=127.15.102.194
--tserver_master_addrs=127.15.102.254:36635
--webserver_port=35451
--webserver_interface=127.15.102.194
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:48:42.690918 32548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:48:42.691208 32548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:48:42.691439 32548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:48:42.698912 32548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:48:42.699028 32548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:48:42.699219 32548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:48:42.709087 32548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:36563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=35451
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:48:42.710415 32548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:48:42.711742 32548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:48:42.725060 32554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:42.725118 32556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:42.725157 32553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:42.725883 32548 server_base.cc:1034] running on GCE node
I20260620 18:48:42.726490 32548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:48:42.727522 32548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:48:42.728717 32548 hybrid_clock.cc:648] HybridClock initialized: now 1781981322728653 us; error 56 us; skew 500 ppm
I20260620 18:48:42.729028 32548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:48:42 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:48:42.732129 32548 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:48:42.733675 32548 webserver.cc:460] Webserver started at http://127.15.102.194:35451/ using document root <none> and password file <none>
I20260620 18:48:42.734370 32548 fs_manager.cc:359] Metadata directory not provided
I20260620 18:48:42.734503 32548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:48:42.740159 32548 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 18:48:42.742892 32563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:42.744257 32548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:48:42.744439 32548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "afe52d9462fd41adb277761ade3a162b"
format_stamp: "Formatted at 2026-06-20 18:47:56 on dist-test-slave-zfp8"
server_key: "af934a0413ec5da09c0ec61b3754ccb5"
server_key_iv: "ae2bf23c00f2d0309e017f58fd6c1194"
server_key_version: "encryptionkey@0"
I20260620 18:48:42.745433 32548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:48:42.766206 32548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:48:42.770277 32548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:48:42.770561 32548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:48:42.771625 32548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:48:42.772972 32548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:48:42.773065 32548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:42.773228 32548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:48:42.773335 32548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:42.807281 32548 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:36563
I20260620 18:48:42.807313 32675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:36563 every 8 connection(s)
I20260620 18:48:42.808790 32548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:48:42.812969 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 32548
I20260620 18:48:42.813648 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:42835
--local_ip_for_outbound_sockets=127.15.102.195
--tserver_master_addrs=127.15.102.254:36635
--webserver_port=36305
--webserver_interface=127.15.102.195
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:48:42.813845 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:42.811834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:43129)
0620 18:48:42.812563 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:42.812850 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":271,"threads_started":1}
W20260620 18:48:42.814471 32676 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:36635 (0 consecutive failures): Network error: Failed to ping master at 127.15.102.254:36635: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
I20260620 18:48:42.815400 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:42.815034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:42079)
0620 18:48:42.815182 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:42.815296 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:48:42.816267 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:42.815966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:33587)
0620 18:48:42.816094 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:42.816163 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 18:48:42.816552 32676 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 18:48:43.141367 32679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:48:43.141703 32679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:48:43.141872 32679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:48:43.149211 32679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:48:43.149366 32679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:48:43.149537 32679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:48:43.159405 32679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:42835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=36305
--tserver_master_addrs=127.15.102.254:36635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:48:43.160818 32679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:48:43.162148 32679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:48:43.175601 32687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:43.175622 32685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:48:43.175889 32684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:48:43.176721 32679 server_base.cc:1034] running on GCE node
I20260620 18:48:43.177467 32679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:48:43.178443 32679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:48:43.179673 32679 hybrid_clock.cc:648] HybridClock initialized: now 1781981323179621 us; error 50 us; skew 500 ppm
I20260620 18:48:43.179999 32679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:48:43 dist-test-slave-zfp8 krb5kdc[31227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981323, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:48:43.183104 32679 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:48:43.184867 32679 webserver.cc:460] Webserver started at http://127.15.102.195:36305/ using document root <none> and password file <none>
I20260620 18:48:43.185576 32679 fs_manager.cc:359] Metadata directory not provided
I20260620 18:48:43.185726 32679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:48:43.191160 32679 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:48:43.193977 32694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:43.195415 32679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:48:43.195590 32679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "b004b3b50aa4474eb66cca64976a3a4b"
format_stamp: "Formatted at 2026-06-20 18:47:57 on dist-test-slave-zfp8"
server_key: "8119525566168b1ea84b47668f347b43"
server_key_iv: "422280abb9f1912e5b9b54b0179b9210"
server_key_version: "encryptionkey@0"
I20260620 18:48:43.196566 32679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:48:43.234807 32679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:48:43.238981 32679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:48:43.239322 32679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:48:43.240348 32679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:48:43.241664 32679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:48:43.241801 32679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:43.241931 32679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:48:43.242024 32679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:48:43.275808 32679 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:42835
I20260620 18:48:43.275831 338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:42835 every 8 connection(s)
I20260620 18:48:43.277307 32679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:48:43.279162 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 32679
I20260620 18:48:43.282877 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:43.280902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:41969)
0620 18:48:43.281722 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:43.281980 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":283,"threads_started":1}
W20260620 18:48:43.283592 339 heartbeater.cc:644] Failed to heartbeat to 127.15.102.254:36635 (0 consecutive failures): Network error: Failed to ping master at 127.15.102.254:36635: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
I20260620 18:48:43.284544 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:43.284188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:53419)
0620 18:48:43.284342 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:43.284413 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 18:48:43.285442 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:43.285080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:33359)
0620 18:48:43.285248 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:43.285331 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 18:48:43.285663 339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 18:48:43.359948 343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:43.359113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:41721)
0620 18:48:43.359640 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:43.359721 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
I20260620 18:48:43.817930 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:43.817101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:41251)
0620 18:48:43.817566 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:43.817655 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260620 18:48:44.287151 345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:44.286304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:59775)
0620 18:48:44.286821 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:44.286924 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":255,"threads_started":1}
I20260620 18:48:44.361966 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:44.361090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:43249)
0620 18:48:44.361653 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:44.361759 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":264,"threads_started":1}
I20260620 18:48:44.819811 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:44.818916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:40507)
0620 18:48:44.819478 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:44.819596 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
I20260620 18:48:45.289232 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:45.288080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:58019)
0620 18:48:45.288891 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:45.289009 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":538,"threads_started":1}
I20260620 18:48:45.363955 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:45.363050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:45479)
0620 18:48:45.363650 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:45.363749 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":354,"threads_started":1}
I20260620 18:48:45.821544 350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:45.820817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:53547)
0620 18:48:45.821267 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:45.821354 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":231,"threads_started":1}
I20260620 18:48:46.290990 351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:46.290167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:38711)
0620 18:48:46.290632 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:46.290721 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":229,"threads_started":1}
I20260620 18:48:46.366106 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:46.365278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:50799)
0620 18:48:46.365793 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:46.365894 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":256,"threads_started":1}
I20260620 18:48:46.823395 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:46.822590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:33125)
0620 18:48:46.823051 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:46.823130 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":235,"threads_started":1}
I20260620 18:48:47.292873 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:47.292022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:55725)
0620 18:48:47.292608 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:47.292697 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":360,"threads_started":1}
I20260620 18:48:47.368368 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:47.367553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:46161)
0620 18:48:47.368056 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:47.368170 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":258,"threads_started":1}
I20260620 18:48:47.825086 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:47.824351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:41739)
0620 18:48:47.824831 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:47.824913 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
I20260620 18:48:48.294632 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:48.293804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:57631)
0620 18:48:48.294359 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:48.294443 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":321,"threads_started":1}
I20260620 18:48:48.370334 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:48.369432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:55701)
0620 18:48:48.369994 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:48.370117 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":290,"threads_started":1}
I20260620 18:48:48.826994 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:48.826051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:59855)
0620 18:48:48.826694 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:48.826797 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":382,"threads_started":1}
I20260620 18:48:49.296406 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:49.295661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:53701)
0620 18:48:49.296136 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:49.296236 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":244,"threads_started":1}
I20260620 18:48:49.372318 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:49.371476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:34359)
0620 18:48:49.372007 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:49.372093 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":256,"threads_started":1}
I20260620 18:48:49.828825 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:49.828039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:46213)
0620 18:48:49.828512 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:49.828612 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":203,"threads_started":1}
I20260620 18:48:50.298249 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:50.297370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:56381)
0620 18:48:50.297944 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:50.298033 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":267,"threads_started":1}
I20260620 18:48:50.374044 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:50.373313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:49123)
0620 18:48:50.373770 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:50.373856 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260620 18:48:50.830924 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:50.829863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:37913)
0620 18:48:50.830474 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:50.830577 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":308,"threads_started":1}
I20260620 18:48:51.300226 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:51.299389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:38295)
0620 18:48:51.299947 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:51.300065 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":328,"threads_started":1}
I20260620 18:48:51.375864 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:51.375081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:46295)
0620 18:48:51.375600 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:51.375685 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":280,"threads_started":1}
I20260620 18:48:51.832722 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:51.831895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:42459)
0620 18:48:51.832448 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:51.832540 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":259,"threads_started":1}
I20260620 18:48:52.301935 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:52.301064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:35967)
0620 18:48:52.301644 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:52.301743 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":296,"threads_started":1}
I20260620 18:48:52.377544 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:52.376797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:60413)
0620 18:48:52.377268 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:52.377351 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":260,"threads_started":1}
I20260620 18:48:52.834438 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:52.833736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:33347)
0620 18:48:52.834196 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:52.834279 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":247,"threads_started":1}
I20260620 18:48:53.303773 372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:53.302916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:37103)
0620 18:48:53.303519 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:53.303607 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":335,"threads_started":1}
I20260620 18:48:53.379349 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:53.378534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:50743)
0620 18:48:53.379021 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:53.379107 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260620 18:48:53.836192 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:53.835442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:44319)
0620 18:48:53.835914 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:53.836011 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":242,"threads_started":1}
I20260620 18:48:54.305452 375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:54.304757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:53385)
0620 18:48:54.305182 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:54.305268 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260620 18:48:54.381079 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:54.380376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:43977)
0620 18:48:54.380823 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:54.380917 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":227,"threads_started":1}
I20260620 18:48:54.838020 377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:54.837277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:40263)
0620 18:48:54.837758 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:54.837858 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":253,"threads_started":1}
I20260620 18:48:55.307178 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:55.306478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:40529)
0620 18:48:55.306931 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:55.307023 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260620 18:48:55.383004 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:55.382168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:34587)
0620 18:48:55.382699 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:55.382824 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":250,"threads_started":1}
I20260620 18:48:55.839840 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:55.838976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:35543)
0620 18:48:55.839547 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:55.839641 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":325,"threads_started":1}
I20260620 18:48:56.308786 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:56.308081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:37833)
0620 18:48:56.308503 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:56.308613 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":215,"threads_started":1}
I20260620 18:48:56.384709 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:56.383922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:38595)
0620 18:48:56.384424 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:56.384519 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":260,"threads_started":1}
I20260620 18:48:56.841449 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:56.840719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:39313)
0620 18:48:56.841175 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:56.841267 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":226,"threads_started":1}
I20260620 18:48:57.310292 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:57.309620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:38801)
0620 18:48:57.310029 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:57.310110 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":229,"threads_started":1}
I20260620 18:48:57.386809 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:57.385988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:53887)
0620 18:48:57.386526 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:57.386634 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260620 18:48:57.843619 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:57.842717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.194:42917)
0620 18:48:57.843309 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:57.843433 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":314,"threads_started":1}
I20260620 18:48:58.290050 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 32377
I20260620 18:48:58.312336 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:58.311416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.195:51807)
0620 18:48:58.311998 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:58.312114 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":279,"threads_started":1}
I20260620 18:48:58.312810 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:58.291414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.15.102.193:35187)
0620 18:48:58.291995 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:58.292018 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:58.292069 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:48:58.293867 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:58.293877 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:58.294332 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:58.294645 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:58.296125 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:58.297397 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:58.298707 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:58.299460 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:58.299604 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:58.306099 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:58.306111 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:58.306135 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:58.306206 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:58.309325 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:58.310025 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:58.310032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:58.310040 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:58.310226 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:58.310552 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:58.310555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:58.310557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:58.311108 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:58.311419 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:58.311969 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 18:48:58.312449 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":282,"threads_started":1}
I20260620 18:48:58.314213 32540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:48:58.401381 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:58.400015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36635 (local address 127.15.102.193:48859)
0620 18:48:58.400643 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0620 18:48:58.400952 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.254:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":323,"threads_started":1}
I20260620 18:48:58.403434 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 32377
I20260620 18:48:58.421639 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 32548
I20260620 18:48:58.439092 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:58.422701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.15.102.194:36563)
0620 18:48:58.423353 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:58.423367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:58.423405 (+ 38us) server_negotiation.cc:407] Connection header received
0620 18:48:58.425057 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:58.425066 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:58.425494 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:58.425840 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:58.427203 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:58.428523 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:58.429880 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:58.430528 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:58.430654 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:58.433398 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:58.433406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:58.433421 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:58.433464 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:58.436228 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:58.436831 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:58.436837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:58.436843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:58.436996 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:58.437325 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:58.437328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:58.437329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:58.437821 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:58.438055 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:58.438377 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 18:48:58.438798 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":314,"threads_started":1}
I20260620 18:48:58.440604 32671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:48:58.531654 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 32548
I20260620 18:48:58.549551 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 32679
I20260620 18:48:58.567648 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:48:58.550652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.15.102.195:42835)
0620 18:48:58.551329 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 18:48:58.551344 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:48:58.551390 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:48:58.553103 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:48:58.553114 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:48:58.553571 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:48:58.553891 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:48:58.555166 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:58.556502 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:58.557762 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:48:58.558447 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:48:58.558584 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:48:58.561203 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:48:58.561210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:48:58.561229 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:48:58.561268 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:48:58.564551 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:58.565172 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:58.565178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:58.565186 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:58.565353 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:48:58.565682 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:48:58.565685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:48:58.565687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:48:58.566222 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:48:58.566454 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:48:58.566811 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 18:48:58.567336 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":315,"threads_started":1}
I20260620 18:48:58.569103 334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:48:58.656206 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 32679
2026-06-20T18:48:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64771 ms)
[----------] 30 tests from SecurityITest (295614 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:48:58 dist-test-slave-zfp8 krb5kdc[397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:48:58 dist-test-slave-zfp8 krb5kdc[397](info): set up 2 sockets
Jun 20 18:48:58 dist-test-slave-zfp8 krb5kdc[397](info): commencing operation
krb5kdc: starting...
W20260620 18:48:59.763770 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:48:59 dist-test-slave-zfp8 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:48:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:00.010645 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40801
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:40801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:00.348709 410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:00.348999 410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:00.349177 410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:00.356331 410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:00.356441 410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:00.356554 410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:00.356652 410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:00.356705 410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:00.366568 410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:40801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:40801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:00.367995 410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:00.369385 410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:00.381961 415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:00.382012 416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:00.381930 418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:00.382962 410 server_base.cc:1034] running on GCE node
I20260620 18:49:00.384009 410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:00.385560 410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:00.386829 410 hybrid_clock.cc:648] HybridClock initialized: now 1781981340386787 us; error 49 us; skew 500 ppm
I20260620 18:49:00.387301 410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:00 dist-test-slave-zfp8 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981340, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:00.390827 410 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:00.392373 410 webserver.cc:460] Webserver started at http://127.15.102.254:34863/ using document root <none> and password file <none>
I20260620 18:49:00.393098 410 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:00.393249 410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:00.393623 410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:00.396736 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "5b17967e0c7940078e1046c1cbbb44eb"
format_stamp: "Formatted at 2026-06-20 18:49:00 on dist-test-slave-zfp8"
server_key: "6c926e7e13964fe0ea93dabafbda028f"
server_key_iv: "fabcef2b0b994a8d8f58c528b25f5d3e"
server_key_version: "encryptionkey@0"
I20260620 18:49:00.397498 410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "5b17967e0c7940078e1046c1cbbb44eb"
format_stamp: "Formatted at 2026-06-20 18:49:00 on dist-test-slave-zfp8"
server_key: "6c926e7e13964fe0ea93dabafbda028f"
server_key_iv: "fabcef2b0b994a8d8f58c528b25f5d3e"
server_key_version: "encryptionkey@0"
I20260620 18:49:00.402748 410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:49:00.406474 425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:00.407887 410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:00.408093 410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "5b17967e0c7940078e1046c1cbbb44eb"
format_stamp: "Formatted at 2026-06-20 18:49:00 on dist-test-slave-zfp8"
server_key: "6c926e7e13964fe0ea93dabafbda028f"
server_key_iv: "fabcef2b0b994a8d8f58c528b25f5d3e"
server_key_version: "encryptionkey@0"
I20260620 18:49:00.408310 410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:00.418988 410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:00.423586 410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:00.423882 410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:00.442677 410 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:40801
I20260620 18:49:00.442698 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:40801 every 8 connection(s)
I20260620 18:49:00.444133 410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:00.444766 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 410
I20260620 18:49:00.445030 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:00.445791 15771 external_mini_cluster.cc:1428] Setting key 46b8445439bc65cac0b9f090d1f028a5
I20260620 18:49:00.449046 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:49:00 dist-test-slave-zfp8 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:00.464465 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb: Bootstrap starting.
I20260620 18:49:00.467893 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:00.469080 477 log.cc:824] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:00.472309 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb: No bootstrap required, opened a new log
I20260620 18:49:00.472956 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:00.447914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.15.102.254:40801)
0620 18:49:00.448743 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:00.448759 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:00.448811 (+ 52us) server_negotiation.cc:407] Connection header received
0620 18:49:00.451131 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:00.451159 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:00.451692 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:00.452181 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:00.454539 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:00.456648 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:00.458598 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:00.459752 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:00.460635 (+ 883us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:00.465323 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:00.465331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:00.465349 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:00.465391 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:00.468576 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:00.469451 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:00.469457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:00.469464 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:00.469637 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:00.470091 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:00.470096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:00.470098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:00.470641 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:00.470943 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:00.471463 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 18:49:00.471898 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":359,"threads_started":1}
I20260620 18:49:00.480394 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b17967e0c7940078e1046c1cbbb44eb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40801 } }
I20260620 18:49:00.480769 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:00.480894 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b17967e0c7940078e1046c1cbbb44eb, State: Initialized, Role: FOLLOWER
I20260620 18:49:00.481465 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b17967e0c7940078e1046c1cbbb44eb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40801 } }
I20260620 18:49:00.481693 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:00.481832 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:00.482002 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:00.484280 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b17967e0c7940078e1046c1cbbb44eb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40801 } }
I20260620 18:49:00.484781 477 leader_election.cc:302] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b17967e0c7940078e1046c1cbbb44eb; no voters:
I20260620 18:49:00.485292 477 leader_election.cc:288] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:00.485457 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:00.486912 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [term 1 LEADER]: Becoming Leader. State: Replica: 5b17967e0c7940078e1046c1cbbb44eb, State: Running, Role: LEADER
I20260620 18:49:00.487555 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:00.487524 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b17967e0c7940078e1046c1cbbb44eb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40801 } }
I20260620 18:49:00.490419 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b17967e0c7940078e1046c1cbbb44eb. Latest consensus state: current_term: 1 leader_uuid: "5b17967e0c7940078e1046c1cbbb44eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b17967e0c7940078e1046c1cbbb44eb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40801 } } }
I20260620 18:49:00.490890 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:00.491477 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b17967e0c7940078e1046c1cbbb44eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b17967e0c7940078e1046c1cbbb44eb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 40801 } } }
I20260620 18:49:00.491830 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:00.491955 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:00.497498 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:00.506847 489 catalog_manager.cc:1347] Generated new cluster ID: 071163632020401c9a8cddc978863ee0
I20260620 18:49:00.507009 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:00.515142 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:00.516465 489 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:00.522929 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb: Generated new TSK 0
I20260620 18:49:00.523806 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:49:00.590703 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40801
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:00.935137 503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:00.935448 503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:00.935607 503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:00.942795 503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:00.942899 503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:00.943104 503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:49:00.952838 503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:00.954196 503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:00.955551 503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:00.968815 508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:00.968782 509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:00.968849 511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:00.969457 503 server_base.cc:1034] running on GCE node
I20260620 18:49:00.970124 503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:00.971212 503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:00.972498 503 hybrid_clock.cc:648] HybridClock initialized: now 1781981340972456 us; error 38 us; skew 500 ppm
I20260620 18:49:00.972846 503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:00 dist-test-slave-zfp8 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981340, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:00.976184 503 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:49:00.977885 503 webserver.cc:460] Webserver started at http://127.15.102.193:36117/ using document root <none> and password file <none>
I20260620 18:49:00.978615 503 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:00.978753 503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:00.979096 503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:00.982151 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "b04b5932ebee448487397138d48fc61a"
format_stamp: "Formatted at 2026-06-20 18:49:00 on dist-test-slave-zfp8"
server_key: "8a2942052826dd8429ab9b4ecf470e36"
server_key_iv: "6f9a272a6a4db824b9407d9e4e1e4b96"
server_key_version: "encryptionkey@0"
I20260620 18:49:00.982959 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "b04b5932ebee448487397138d48fc61a"
format_stamp: "Formatted at 2026-06-20 18:49:00 on dist-test-slave-zfp8"
server_key: "8a2942052826dd8429ab9b4ecf470e36"
server_key_iv: "6f9a272a6a4db824b9407d9e4e1e4b96"
server_key_version: "encryptionkey@0"
I20260620 18:49:00.988580 503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 18:49:00.992533 518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:00.994107 503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:49:00.994309 503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "b04b5932ebee448487397138d48fc61a"
format_stamp: "Formatted at 2026-06-20 18:49:00 on dist-test-slave-zfp8"
server_key: "8a2942052826dd8429ab9b4ecf470e36"
server_key_iv: "6f9a272a6a4db824b9407d9e4e1e4b96"
server_key_version: "encryptionkey@0"
I20260620 18:49:00.994530 503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:01.011561 503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:01.016006 503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:01.016312 503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:01.017334 503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:01.018760 503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:01.018833 503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:01.019016 503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:01.019068 503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:01.051983 503 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:34323
I20260620 18:49:01.052035 630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:34323 every 8 connection(s)
I20260620 18:49:01.053364 503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:49:01.060474 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 503
I20260620 18:49:01.060696 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:49:01.061362 15771 external_mini_cluster.cc:1428] Setting key a003682f020cf7ae0381b164e56d241c
Jun 20 18:49:01 dist-test-slave-zfp8 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981340, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:01.076408 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:01.056097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45459 (local address 127.15.102.254:40801)
0620 18:49:01.056613 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:01.056625 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:01.057478 (+ 853us) server_negotiation.cc:407] Connection header received
0620 18:49:01.059546 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:01.059552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:01.059656 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:01.059790 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:01.061618 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:01.062601 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:01.064705 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:01.065484 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:01.065644 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:01.069444 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:01.069453 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:01.069461 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:01.069507 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:01.072726 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:01.073798 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:01.073802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:01.073804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:01.073925 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:01.074465 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:01.074467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:01.074469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:01.074737 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:01.074926 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:01.075949 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0620 18:49:01.076130 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":276,"threads_started":1}
I20260620 18:49:01.078239 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:01.056579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40801 (local address 127.15.102.193:45459)
0620 18:49:01.057309 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:01.057356 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:01.059197 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:01.060051 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:01.060060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:01.060701 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:01.061352 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:01.061370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:01.062797 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:01.062803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:01.064542 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:01.064555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:01.065747 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:01.065752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:01.065974 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:01.066771 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:01.066796 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:01.069196 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:01.072915 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:01.072928 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:01.072949 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:01.073648 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:01.074046 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:01.074054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:01.074061 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:01.074342 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:01.074928 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:01.074940 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:01.075499 (+ 559us) client_negotiation.cc:769] Sending connection context
0620 18:49:01.075870 (+ 371us) client_negotiation.cc:240] Negotiation successful
0620 18:49:01.076500 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":274,"threads_started":1}
I20260620 18:49:01.079995 631 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40801
I20260620 18:49:01.080330 631 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:01.081085 631 heartbeater.cc:511] Master 127.15.102.254:40801 requested a full tablet report, sending...
I20260620 18:49:01.083395 442 ts_manager.cc:194] Registered new tserver with Master: b04b5932ebee448487397138d48fc61a (127.15.102.193:34323)
I20260620 18:49:01.084986 442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:45459
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:49:01.118779 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40801
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:01.450937 637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:01.451277 637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:01.451448 637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:01.458731 637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:01.458889 637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:01.459064 637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:49:01.468648 637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:01.469944 637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:01.471340 637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:01.484467 643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:01.484509 642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:01.484532 645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:01.485661 637 server_base.cc:1034] running on GCE node
I20260620 18:49:01.486236 637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:01.487110 637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:01.488315 637 hybrid_clock.cc:648] HybridClock initialized: now 1781981341488270 us; error 46 us; skew 500 ppm
I20260620 18:49:01.488658 637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:01 dist-test-slave-zfp8 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981341, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:01.491868 637 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:49:01.493485 637 webserver.cc:460] Webserver started at http://127.15.102.194:38277/ using document root <none> and password file <none>
I20260620 18:49:01.494180 637 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:01.494318 637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:01.494650 637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:01.497664 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "0348b207f601497b865017c79b9abc52"
format_stamp: "Formatted at 2026-06-20 18:49:01 on dist-test-slave-zfp8"
server_key: "78f21ca65119d7e06a1db9aab04ac759"
server_key_iv: "f99156c5965aad2318ad15cc0a712906"
server_key_version: "encryptionkey@0"
I20260620 18:49:01.498466 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "0348b207f601497b865017c79b9abc52"
format_stamp: "Formatted at 2026-06-20 18:49:01 on dist-test-slave-zfp8"
server_key: "78f21ca65119d7e06a1db9aab04ac759"
server_key_iv: "f99156c5965aad2318ad15cc0a712906"
server_key_version: "encryptionkey@0"
I20260620 18:49:01.503672 637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 18:49:01.507177 652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:01.508569 637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:01.508737 637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "0348b207f601497b865017c79b9abc52"
format_stamp: "Formatted at 2026-06-20 18:49:01 on dist-test-slave-zfp8"
server_key: "78f21ca65119d7e06a1db9aab04ac759"
server_key_iv: "f99156c5965aad2318ad15cc0a712906"
server_key_version: "encryptionkey@0"
I20260620 18:49:01.508942 637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:01.542723 637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:01.547022 637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:01.547364 637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:01.548398 637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:01.549765 637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:01.549840 637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:01.550024 637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:01.550074 637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:01.583953 637 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:35531
I20260620 18:49:01.584033 764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:35531 every 8 connection(s)
I20260620 18:49:01.585327 637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:49:01.594079 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 637
I20260620 18:49:01.594290 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:49:01.594854 15771 external_mini_cluster.cc:1428] Setting key 52d8368c7b33fdca403793809a60ed73
Jun 20 18:49:01 dist-test-slave-zfp8 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981341, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:01.607506 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:01.588474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:44833 (local address 127.15.102.254:40801)
0620 18:49:01.588947 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:01.588951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:01.590019 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 18:49:01.592002 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:01.592008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:01.592102 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:01.592223 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:01.594035 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:01.595026 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:01.596982 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:01.597421 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:01.597563 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:01.601678 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:01.601684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:01.601687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:01.601721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:01.604407 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:01.605142 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:01.605149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:01.605158 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:01.605289 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:01.605728 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:01.605737 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:01.605739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:01.606020 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:01.606192 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:01.606883 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 18:49:01.607110 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":246,"threads_started":1}
I20260620 18:49:01.608506 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:01.589086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40801 (local address 127.15.102.194:44833)
0620 18:49:01.589854 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:01.589899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:01.591699 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:01.592479 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:01.592487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:01.593048 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:01.593819 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:01.593850 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:01.595179 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:01.595184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:01.596835 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:01.596847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:01.597721 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:01.597732 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:01.597960 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:01.598719 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:01.598774 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:01.601435 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:01.604557 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:01.604567 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:01.604584 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:01.605010 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:01.605403 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:01.605408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:01.605412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:01.605600 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:01.606151 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:01.606159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:01.606511 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 18:49:01.606781 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 18:49:01.607330 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":316,"threads_started":1}
I20260620 18:49:01.609997 765 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40801
I20260620 18:49:01.610338 765 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:01.611224 765 heartbeater.cc:511] Master 127.15.102.254:40801 requested a full tablet report, sending...
I20260620 18:49:01.612883 442 ts_manager.cc:194] Registered new tserver with Master: 0348b207f601497b865017c79b9abc52 (127.15.102.194:35531)
I20260620 18:49:01.614161 442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:44833
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:49:01.649801 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40801
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:43637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:01.988111 771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:01.988399 771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:01.988494 771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:01.996001 771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:01.996119 771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:01.996243 771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:49:02.006170 771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:43637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:40801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:02.007519 771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:02.008843 771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:02.022040 776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:02.022078 777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:02.022118 779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:02.023222 771 server_base.cc:1034] running on GCE node
I20260620 18:49:02.023874 771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:02.024902 771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:02.026144 771 hybrid_clock.cc:648] HybridClock initialized: now 1781981342026099 us; error 41 us; skew 500 ppm
I20260620 18:49:02.026476 771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:02 dist-test-slave-zfp8 krb5kdc[397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981342, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:02.029774 771 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:49:02.031320 771 webserver.cc:460] Webserver started at http://127.15.102.195:33117/ using document root <none> and password file <none>
I20260620 18:49:02.032038 771 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:02.032195 771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:02.032516 771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:02.035737 771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "19ae0da907c646a8b628def4280e0ac1"
format_stamp: "Formatted at 2026-06-20 18:49:02 on dist-test-slave-zfp8"
server_key: "c6dcf2ea6225fc705104f4dfd11c25f8"
server_key_iv: "d08405cba01a90d38b5a8b0fb8579bc2"
server_key_version: "encryptionkey@0"
I20260620 18:49:02.036548 771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "19ae0da907c646a8b628def4280e0ac1"
format_stamp: "Formatted at 2026-06-20 18:49:02 on dist-test-slave-zfp8"
server_key: "c6dcf2ea6225fc705104f4dfd11c25f8"
server_key_iv: "d08405cba01a90d38b5a8b0fb8579bc2"
server_key_version: "encryptionkey@0"
I20260620 18:49:02.041841 771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:49:02.045416 786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:02.046789 771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:49:02.046977 771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "19ae0da907c646a8b628def4280e0ac1"
format_stamp: "Formatted at 2026-06-20 18:49:02 on dist-test-slave-zfp8"
server_key: "c6dcf2ea6225fc705104f4dfd11c25f8"
server_key_iv: "d08405cba01a90d38b5a8b0fb8579bc2"
server_key_version: "encryptionkey@0"
I20260620 18:49:02.047197 771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:02.061291 771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:02.065692 771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:02.065994 771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:02.066953 771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:02.068315 771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:02.068418 771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:02.068563 771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:02.068648 771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:02.088348 631 heartbeater.cc:503] Master 127.15.102.254:40801 was elected leader, sending a full tablet report...
I20260620 18:49:02.102458 771 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:33941
I20260620 18:49:02.102560 899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:33941 every 8 connection(s)
I20260620 18:49:02.103852 771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:49:02.113623 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 771
I20260620 18:49:02.113806 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:49:02.114261 15771 external_mini_cluster.cc:1428] Setting key ecf6d8c0480fd65a7b2edef5fb360fd2
Jun 20 18:49:02 dist-test-slave-zfp8 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981342, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:02.124320 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.106586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:44685 (local address 127.15.102.254:40801)
0620 18:49:02.106826 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.106834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.108132 (+ 1298us) server_negotiation.cc:407] Connection header received
0620 18:49:02.110082 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.110087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.110167 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.110270 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:02.111897 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.112723 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.114546 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.115066 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.115173 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.118763 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:02.118769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:02.118773 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:02.118817 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:02.121286 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.121999 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.122005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.122010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.122138 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.122565 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.122579 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.122581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.122845 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:02.123006 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.123756 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.123984 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 18:49:02.125393 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.107170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:40801 (local address 127.15.102.195:44685)
0620 18:49:02.107961 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:02.108001 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:02.109801 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:02.110477 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:02.110485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:02.111015 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:02.111693 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.111732 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.112918 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.112926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.114397 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.114416 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.115314 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.115324 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.115571 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.116330 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:02.116353 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:02.118565 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:02.121435 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:02.121444 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:02.121461 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:02.121871 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:02.122250 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:02.122255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:02.122258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:02.122442 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:02.122982 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:02.122991 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:02.123394 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 18:49:02.123655 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 18:49:02.124192 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":298,"threads_started":1}
I20260620 18:49:02.126739 900 heartbeater.cc:347] Connected to a master server at 127.15.102.254:40801
I20260620 18:49:02.127090 900 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:02.127907 900 heartbeater.cc:511] Master 127.15.102.254:40801 requested a full tablet report, sending...
I20260620 18:49:02.129563 442 ts_manager.cc:194] Registered new tserver with Master: 19ae0da907c646a8b628def4280e0ac1 (127.15.102.195:33941)
I20260620 18:49:02.130271 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:49:02.130578 442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:44685
I20260620 18:49:02.148193 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.135032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.15.102.254:40801)
0620 18:49:02.135195 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.135200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.135688 (+ 488us) server_negotiation.cc:407] Connection header received
0620 18:49:02.135824 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.135828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.135895 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.135977 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:02.137391 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.138252 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.139761 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.140243 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.140362 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.142753 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:02.142758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:02.142762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:02.142807 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:02.145685 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.146295 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.146299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.146301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.146450 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.146819 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.146823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.146825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.147174 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:02.147489 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.147754 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.147921 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:49:02.152581 442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:49:02.159592 442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:49:02.187579 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.172241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:33941 (local address 127.0.0.1:48028)
0620 18:49:02.173037 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:02.173083 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:02.173284 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:02.174207 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:02.174216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:02.174248 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:02.174648 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.174662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.180298 (+ 5636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.180307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.183348 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.183371 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.186416 (+ 3045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.186426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.186673 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.186732 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 18:49:02.186889 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:49:02.187121 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":368,"threads_started":1}
I20260620 18:49:02.187760 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.172411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.15.102.195:33941)
0620 18:49:02.173527 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.173537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.173571 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:49:02.173674 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.173682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.173960 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.174871 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:02.176657 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.178832 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.184262 (+ 5430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.186126 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.186419 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.186665 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.186952 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.187203 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":791,"threads_started":1}
I20260620 18:49:02.193221 835 tablet_service.cc:1467] Processing CreateTablet for tablet d17b787ad3444c7398396e4024266c0e (DEFAULT_TABLE table=test-table [id=6d0d6f81ba90414a958af6883a643ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:02.194583 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.173543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34323 (local address 127.0.0.1:55636)
0620 18:49:02.174065 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:02.174083 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:02.174190 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:02.184485 (+ 10295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:02.184491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:02.184520 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:02.184883 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.184894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.189772 (+ 4878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.189780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.191916 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.191932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.193657 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.193663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.193858 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.193898 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:49:02.193991 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:49:02.194138 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":275,"threads_started":1}
I20260620 18:49:02.194784 835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b787ad3444c7398396e4024266c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:02.195479 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.173947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.15.102.193:34323)
0620 18:49:02.183797 (+ 9850us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.183807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.183846 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:49:02.183962 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.183968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.184291 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.184521 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:02.188159 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.189643 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.192080 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.193437 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.193657 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.193849 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.194957 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.195126 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9646,"thread_start_us":9489,"threads_started":1}
I20260620 18:49:02.199720 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.174127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35531 (local address 127.0.0.1:43638)
0620 18:49:02.182853 (+ 8726us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:02.182874 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:02.183001 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:02.191342 (+ 8341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:02.191351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:02.191391 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:02.191968 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.191982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.195714 (+ 3732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.195736 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.197359 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.197373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.198897 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.198903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.199077 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.199102 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:49:02.199219 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 18:49:02.199387 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8536,"thread_start_us":8961,"threads_started":1}
I20260620 18:49:02.199757 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.174669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.15.102.194:35531)
0620 18:49:02.190669 (+ 16000us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.190678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.190708 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:49:02.190829 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.190837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.191083 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.191320 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:02.193941 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.195530 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.197614 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.198593 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.198798 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.198982 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.199298 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.199453 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15814,"thread_start_us":15942,"threads_started":1}
I20260620 18:49:02.199473 566 tablet_service.cc:1467] Processing CreateTablet for tablet d17b787ad3444c7398396e4024266c0e (DEFAULT_TABLE table=test-table [id=6d0d6f81ba90414a958af6883a643ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:02.201485 566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b787ad3444c7398396e4024266c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:02.202963 700 tablet_service.cc:1467] Processing CreateTablet for tablet d17b787ad3444c7398396e4024266c0e (DEFAULT_TABLE table=test-table [id=6d0d6f81ba90414a958af6883a643ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:02.204291 700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b787ad3444c7398396e4024266c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:02.209820 918 tablet_bootstrap.cc:492] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Bootstrap starting.
I20260620 18:49:02.213929 918 tablet_bootstrap.cc:654] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:02.215220 918 log.cc:824] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:02.215710 919 tablet_bootstrap.cc:492] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Bootstrap starting.
I20260620 18:49:02.219137 919 tablet_bootstrap.cc:654] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:02.217850 920 tablet_bootstrap.cc:492] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Bootstrap starting.
I20260620 18:49:02.219581 918 tablet_bootstrap.cc:492] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: No bootstrap required, opened a new log
I20260620 18:49:02.219852 918 ts_tablet_manager.cc:1397] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:49:02.220425 919 log.cc:824] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:02.222699 920 tablet_bootstrap.cc:654] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:02.223999 919 tablet_bootstrap.cc:492] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: No bootstrap required, opened a new log
I20260620 18:49:02.224225 920 log.cc:824] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:02.224299 919 ts_tablet_manager.cc:1397] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 18:49:02.227355 920 tablet_bootstrap.cc:492] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: No bootstrap required, opened a new log
I20260620 18:49:02.227634 920 ts_tablet_manager.cc:1397] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260620 18:49:02.228649 918 raft_consensus.cc:348] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.229049 918 raft_consensus.cc:374] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:02.229153 918 raft_consensus.cc:729] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19ae0da907c646a8b628def4280e0ac1, State: Initialized, Role: FOLLOWER
I20260620 18:49:02.229811 918 consensus_queue.cc:260] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.230917 900 heartbeater.cc:503] Master 127.15.102.254:40801 was elected leader, sending a full tablet report...
I20260620 18:49:02.233143 918 ts_tablet_manager.cc:1428] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260620 18:49:02.235026 919 raft_consensus.cc:348] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.235528 919 raft_consensus.cc:374] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:02.235618 919 raft_consensus.cc:729] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04b5932ebee448487397138d48fc61a, State: Initialized, Role: FOLLOWER
I20260620 18:49:02.236353 919 consensus_queue.cc:260] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.238255 920 raft_consensus.cc:348] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.238723 920 raft_consensus.cc:374] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:02.238830 920 raft_consensus.cc:729] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0348b207f601497b865017c79b9abc52, State: Initialized, Role: FOLLOWER
I20260620 18:49:02.239553 919 ts_tablet_manager.cc:1428] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 18:49:02.239466 920 consensus_queue.cc:260] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.241499 765 heartbeater.cc:503] Master 127.15.102.254:40801 was elected leader, sending a full tablet report...
I20260620 18:49:02.242005 920 ts_tablet_manager.cc:1428] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
W20260620 18:49:02.305814 632 tablet.cc:2369] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:49:02.337780 766 tablet.cc:2369] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:49:02.355687 901 tablet.cc:2369] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:02.544472 924 raft_consensus.cc:482] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:49:02.544847 924 raft_consensus.cc:504] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.546612 924 leader_election.cc:288] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b04b5932ebee448487397138d48fc61a (127.15.102.193:34323), 0348b207f601497b865017c79b9abc52 (127.15.102.194:35531)
I20260620 18:49:02.553983 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.547105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:33565 (local address 127.15.102.193:34323)
0620 18:49:02.547382 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.547389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.547413 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:49:02.547495 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.547508 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.547581 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.547703 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:02.548299 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.549340 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.551873 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.553097 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.553270 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.553353 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.553646 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.553754 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:49:02.554056 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.546950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34323 (local address 127.15.102.195:33565)
0620 18:49:02.547144 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:02.547167 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:02.547338 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:02.547902 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:02.547906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:02.547935 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:02.548135 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.548147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.549503 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.549508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.551694 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.551708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.553269 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.553274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.553422 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.553448 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 18:49:02.553608 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 18:49:02.553750 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":47}
I20260620 18:49:02.554546 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.547412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:35531 (local address 127.15.102.195:44541)
0620 18:49:02.548030 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:02.548048 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:02.548167 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:02.548510 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:02.548512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:02.548531 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:02.548797 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.548803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.549966 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.549977 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.551965 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:02.551973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.553613 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.553620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:02.553780 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.553808 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 18:49:02.553904 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 18:49:02.554012 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":24,"thread_start_us":338,"threads_started":1}
I20260620 18:49:02.554629 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.547757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:44541 (local address 127.15.102.194:35531)
0620 18:49:02.547967 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.547972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.548133 (+ 161us) server_negotiation.cc:407] Connection header received
0620 18:49:02.548298 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.548303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.548385 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.548486 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:02.548942 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.549796 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.552108 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.553397 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.553523 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.553616 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.553941 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.554090 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:49:02.555160 586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b787ad3444c7398396e4024266c0e" candidate_uuid: "19ae0da907c646a8b628def4280e0ac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04b5932ebee448487397138d48fc61a" is_pre_election: true
I20260620 18:49:02.555639 720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b787ad3444c7398396e4024266c0e" candidate_uuid: "19ae0da907c646a8b628def4280e0ac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0348b207f601497b865017c79b9abc52" is_pre_election: true
I20260620 18:49:02.555735 586 raft_consensus.cc:2454] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ae0da907c646a8b628def4280e0ac1 in term 0.
I20260620 18:49:02.556095 720 raft_consensus.cc:2454] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19ae0da907c646a8b628def4280e0ac1 in term 0.
I20260620 18:49:02.556437 787 leader_election.cc:302] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ae0da907c646a8b628def4280e0ac1, b04b5932ebee448487397138d48fc61a; no voters:
I20260620 18:49:02.556842 924 raft_consensus.cc:2781] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:49:02.557003 924 raft_consensus.cc:482] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:49:02.557125 924 raft_consensus.cc:3037] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:02.559399 924 raft_consensus.cc:504] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.559973 924 leader_election.cc:288] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [CANDIDATE]: Term 1 election: Requested vote from peers b04b5932ebee448487397138d48fc61a (127.15.102.193:34323), 0348b207f601497b865017c79b9abc52 (127.15.102.194:35531)
I20260620 18:49:02.560499 586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b787ad3444c7398396e4024266c0e" candidate_uuid: "19ae0da907c646a8b628def4280e0ac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04b5932ebee448487397138d48fc61a"
I20260620 18:49:02.560549 720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b787ad3444c7398396e4024266c0e" candidate_uuid: "19ae0da907c646a8b628def4280e0ac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0348b207f601497b865017c79b9abc52"
I20260620 18:49:02.560732 586 raft_consensus.cc:3037] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:02.560794 720 raft_consensus.cc:3037] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:02.562950 586 raft_consensus.cc:2454] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ae0da907c646a8b628def4280e0ac1 in term 1.
I20260620 18:49:02.562959 720 raft_consensus.cc:2454] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19ae0da907c646a8b628def4280e0ac1 in term 1.
I20260620 18:49:02.563380 787 leader_election.cc:302] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19ae0da907c646a8b628def4280e0ac1, b04b5932ebee448487397138d48fc61a; no voters:
I20260620 18:49:02.563694 924 raft_consensus.cc:2781] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:02.564148 924 raft_consensus.cc:686] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 1 LEADER]: Becoming Leader. State: Replica: 19ae0da907c646a8b628def4280e0ac1, State: Running, Role: LEADER
I20260620 18:49:02.564728 924 consensus_queue.cc:237] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } }
I20260620 18:49:02.569380 442 catalog_manager.cc:5495] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19ae0da907c646a8b628def4280e0ac1 (127.15.102.195). New cstate: current_term: 1 leader_uuid: "19ae0da907c646a8b628def4280e0ac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19ae0da907c646a8b628def4280e0ac1" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 33941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 } health_report { overall_health: UNKNOWN } } }
I20260620 18:49:02.638980 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.633338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.15.102.195:33941)
0620 18:49:02.633552 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.633558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.633634 (+ 76us) server_negotiation.cc:407] Connection header received
0620 18:49:02.633897 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.633901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.633982 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.634141 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:02.634529 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.635326 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.636784 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.637188 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.637336 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.637864 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:02.638399 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:02.638576 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.638627 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.638711 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:49:02.647544 720 raft_consensus.cc:1261] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 1 FOLLOWER]: Refusing update from remote peer 19ae0da907c646a8b628def4280e0ac1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:02.647574 586 raft_consensus.cc:1261] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 1 FOLLOWER]: Refusing update from remote peer 19ae0da907c646a8b628def4280e0ac1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:02.648500 924 consensus_queue.cc:1035] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0348b207f601497b865017c79b9abc52" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:02.648857 928 consensus_queue.cc:1035] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b04b5932ebee448487397138d48fc61a" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:02.676240 441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:02.676525 441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:02.680341 441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b17967e0c7940078e1046c1cbbb44eb: Sending DeleteTablet for 3 replicas of tablet d17b787ad3444c7398396e4024266c0e
I20260620 18:49:02.681741 835 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b787ad3444c7398396e4024266c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:02 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:48028
I20260620 18:49:02.681787 700 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b787ad3444c7398396e4024266c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:02 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:43638
I20260620 18:49:02.681816 566 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b787ad3444c7398396e4024266c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:02 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:55636
I20260620 18:49:02.683964 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 503
I20260620 18:49:02.693764 941 tablet_replica.cc:330] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: stopping tablet replica
I20260620 18:49:02.694475 941 raft_consensus.cc:2229] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:49:02.695358 941 raft_consensus.cc:2258] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:02.696185 942 tablet_replica.cc:330] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: stopping tablet replica
I20260620 18:49:02.696622 942 raft_consensus.cc:2229] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:02.697144 942 raft_consensus.cc:2258] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:02.699368 941 ts_tablet_manager.cc:1905] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:02.700109 942 ts_tablet_manager.cc:1905] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:02.706712 941 ts_tablet_manager.cc:1918] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:02.706890 941 log.cc:1196] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/d17b787ad3444c7398396e4024266c0e
I20260620 18:49:02.707481 941 ts_tablet_manager.cc:1939] T d17b787ad3444c7398396e4024266c0e P 19ae0da907c646a8b628def4280e0ac1: Deleting consensus metadata
I20260620 18:49:02.709349 426 catalog_manager.cc:4841] TS 19ae0da907c646a8b628def4280e0ac1 (127.15.102.195:33941): tablet d17b787ad3444c7398396e4024266c0e (table test-table [id=6d0d6f81ba90414a958af6883a643ead]) successfully deleted
I20260620 18:49:02.710052 942 ts_tablet_manager.cc:1918] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:02.710140 943 tablet_replica.cc:330] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: stopping tablet replica
I20260620 18:49:02.710332 942 log.cc:1196] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/d17b787ad3444c7398396e4024266c0e
I20260620 18:49:02.710624 943 raft_consensus.cc:2229] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:02.711004 942 ts_tablet_manager.cc:1939] T d17b787ad3444c7398396e4024266c0e P b04b5932ebee448487397138d48fc61a: Deleting consensus metadata
I20260620 18:49:02.711151 943 raft_consensus.cc:2258] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:02.713464 943 ts_tablet_manager.cc:1905] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:02.714967 426 catalog_manager.cc:4841] TS b04b5932ebee448487397138d48fc61a (127.15.102.193:34323): tablet d17b787ad3444c7398396e4024266c0e (table test-table [id=6d0d6f81ba90414a958af6883a643ead]) successfully deleted
Jun 20 18:49:02 dist-test-slave-zfp8 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:49:02.720471 943 ts_tablet_manager.cc:1918] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:02.720649 943 log.cc:1196] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/d17b787ad3444c7398396e4024266c0e
I20260620 18:49:02.721187 943 ts_tablet_manager.cc:1939] T d17b787ad3444c7398396e4024266c0e P 0348b207f601497b865017c79b9abc52: Deleting consensus metadata
I20260620 18:49:02.722716 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.685740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.15.102.193:34323)
0620 18:49:02.685973 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.685982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.695418 (+ 9436us) server_negotiation.cc:407] Connection header received
0620 18:49:02.695638 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.695646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.695768 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.695932 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:02.697697 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.698846 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.710051 (+ 11205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.710831 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.710978 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.716471 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:02.716483 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:02.716497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:02.716537 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:02.719673 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.720410 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.720417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.720419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.720548 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.721036 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.721041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.721044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.721466 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:02.721785 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.722102 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.722319 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:49:02.723420 426 catalog_manager.cc:4841] TS 0348b207f601497b865017c79b9abc52 (127.15.102.194:35531): tablet d17b787ad3444c7398396e4024266c0e (table test-table [id=6d0d6f81ba90414a958af6883a643ead]) successfully deleted
I20260620 18:49:02.723636 626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:02.814841 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 503
I20260620 18:49:02.839205 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 637
Jun 20 18:49:02 dist-test-slave-zfp8 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:49:02.857208 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.840252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.15.102.194:35531)
0620 18:49:02.840426 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.840430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.840449 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:49:02.840578 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.840582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.840655 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.840741 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:02.842392 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.843110 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.845108 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.845516 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.845610 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.850789 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:02.850815 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:02.850839 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:02.850891 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:02.854114 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.854864 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.854874 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.854878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.855016 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.855509 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.855515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.855519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.855906 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:02.856236 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:02.856517 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:49:02.856742 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:49:02.857854 760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:02.959329 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 637
I20260620 18:49:02.983111 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 771
Jun 20 18:49:02 dist-test-slave-zfp8 krb5kdc[397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:49:03.001964 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:02.983979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.15.102.195:33941)
0620 18:49:02.984345 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:02.984350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:02.984364 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:49:02.984415 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:02.984419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:02.984510 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:02.984588 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:02.986364 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.987146 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.989095 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:02.989611 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:02.989716 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:02.995652 (+ 5936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:02.995666 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:02.995684 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:02.995730 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:02.998835 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:02.999746 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:02.999754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:02.999756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:02.999908 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:03.000368 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:03.000372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:03.000374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:03.000780 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:03.001005 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:03.001418 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 18:49:03.001588 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260620 18:49:03.002753 895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:03.126446 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 771
I20260620 18:49:03.146277 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 410
I20260620 18:49:03.147490 472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:03.257982 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 410
2026-06-20T18:49:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4605 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T18:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:03Z Disabled control of system clock
I20260620 18:49:03.336280 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35963
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:35963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:49:03.668802 952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:03.669112 952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:03.669251 952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:03.676595 952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:03.676707 952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:03.676782 952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:03.676888 952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:03.676950 952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:03.686362 952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:35963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:03.687717 952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:03.689621 952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:03.701508 957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:03.701426 958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:03.701465 960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:03.702346 952 server_base.cc:1034] running on GCE node
I20260620 18:49:03.703408 952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:03.704808 952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:03.706067 952 hybrid_clock.cc:648] HybridClock initialized: now 1781981343705988 us; error 45 us; skew 500 ppm
I20260620 18:49:03.706425 952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:03.714121 952 webserver.cc:460] Webserver started at http://127.15.102.254:37543/ using document root <none> and password file <none>
I20260620 18:49:03.714875 952 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:03.715018 952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:03.715428 952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:03.718382 952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9"
format_stamp: "Formatted at 2026-06-20 18:49:03 on dist-test-slave-zfp8"
server_key: "a77246720f782e247351a58a5f9598dd"
server_key_iv: "3b78c52e0d882ffc51df1759aeef5c53"
server_key_version: "encryptionkey@0"
I20260620 18:49:03.719205 952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9"
format_stamp: "Formatted at 2026-06-20 18:49:03 on dist-test-slave-zfp8"
server_key: "a77246720f782e247351a58a5f9598dd"
server_key_iv: "3b78c52e0d882ffc51df1759aeef5c53"
server_key_version: "encryptionkey@0"
I20260620 18:49:03.724421 952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:49:03.728173 966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:03.729708 952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:03.729861 952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9"
format_stamp: "Formatted at 2026-06-20 18:49:03 on dist-test-slave-zfp8"
server_key: "a77246720f782e247351a58a5f9598dd"
server_key_iv: "3b78c52e0d882ffc51df1759aeef5c53"
server_key_version: "encryptionkey@0"
I20260620 18:49:03.730057 952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:03.751801 952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:03.752632 952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:03.752907 952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:03.771842 952 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:35963
I20260620 18:49:03.771849 1017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:35963 every 8 connection(s)
I20260620 18:49:03.773324 952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:03.777288 1018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:03.779706 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 952
I20260620 18:49:03.779896 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:03.780429 15771 external_mini_cluster.cc:1428] Setting key 8d586c582552040e597b8fa075bfb2f7
I20260620 18:49:03.790007 1018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9: Bootstrap starting.
I20260620 18:49:03.793696 1018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:03.794710 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:03.782483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.15.102.254:35963)
0620 18:49:03.783731 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:03.783742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:03.783785 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:49:03.785433 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:03.785455 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:03.785913 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:03.786256 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:03.786843 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:03.788252 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:03.790259 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:03.790934 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:03.791071 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:03.792349 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:03.792356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:03.792370 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:03.792918 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:03.793012 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:03.793033 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:03.793122 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:03.793353 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:49:03.793700 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":534,"threads_started":1}
I20260620 18:49:03.794744 1018 log.cc:824] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:03.797777 1018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9: No bootstrap required, opened a new log
I20260620 18:49:03.805482 1018 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35963 } }
I20260620 18:49:03.805866 1018 raft_consensus.cc:374] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:03.805965 1018 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13d01e8c97f401da72ea6fe5d6e1bf9, State: Initialized, Role: FOLLOWER
I20260620 18:49:03.806594 1018 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35963 } }
I20260620 18:49:03.806834 1018 raft_consensus.cc:388] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:03.806968 1018 raft_consensus.cc:482] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:03.807144 1018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:03.809635 1018 raft_consensus.cc:504] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35963 } }
I20260620 18:49:03.810161 1018 leader_election.cc:302] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f13d01e8c97f401da72ea6fe5d6e1bf9; no voters:
I20260620 18:49:03.810756 1018 leader_election.cc:288] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:03.810921 1023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:03.812176 1023 raft_consensus.cc:686] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [term 1 LEADER]: Becoming Leader. State: Replica: f13d01e8c97f401da72ea6fe5d6e1bf9, State: Running, Role: LEADER
I20260620 18:49:03.812739 1023 consensus_queue.cc:237] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35963 } }
I20260620 18:49:03.813076 1018 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:03.815824 1025 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f13d01e8c97f401da72ea6fe5d6e1bf9. Latest consensus state: current_term: 1 leader_uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35963 } } }
I20260620 18:49:03.815871 1024 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f13d01e8c97f401da72ea6fe5d6e1bf9" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35963 } } }
I20260620 18:49:03.816215 1025 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:03.816262 1024 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:03.817243 1032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:03.822854 1032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:03.832755 1032 catalog_manager.cc:1347] Generated new cluster ID: 47c8eaf73c8144898af6b93bd113d643
I20260620 18:49:03.832892 1032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:03.866636 1032 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:03.868505 1032 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:03.881949 1032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9: Generated new TSK 0
I20260620 18:49:03.883015 1032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:03.893500 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35963
--builtin_ntp_servers=127.15.102.212:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:49:04.231370 1042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:04.231774 1042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:04.231937 1042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:04.239408 1042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:04.239558 1042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:04.239744 1042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:49:04.248843 1042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:04.250226 1042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:04.252081 1042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:04.265698 1048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:04.265779 1050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:04.265733 1047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:04.266481 1042 server_base.cc:1034] running on GCE node
I20260620 18:49:04.267197 1042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:04.268308 1042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:04.269511 1042 hybrid_clock.cc:648] HybridClock initialized: now 1781981344269452 us; error 65 us; skew 500 ppm
I20260620 18:49:04.269836 1042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:04.272472 1042 webserver.cc:460] Webserver started at http://127.15.102.193:37627/ using document root <none> and password file <none>
I20260620 18:49:04.273306 1042 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:04.273420 1042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:04.273751 1042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:04.276721 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "ff8536b6644b4891bdb9d6e424b89080"
format_stamp: "Formatted at 2026-06-20 18:49:04 on dist-test-slave-zfp8"
server_key: "9f56692c45881b0f07a7c0c06e8b15a9"
server_key_iv: "83ec3386c32181c59e88d1adef59ba8a"
server_key_version: "encryptionkey@0"
I20260620 18:49:04.277519 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "ff8536b6644b4891bdb9d6e424b89080"
format_stamp: "Formatted at 2026-06-20 18:49:04 on dist-test-slave-zfp8"
server_key: "9f56692c45881b0f07a7c0c06e8b15a9"
server_key_iv: "83ec3386c32181c59e88d1adef59ba8a"
server_key_version: "encryptionkey@0"
I20260620 18:49:04.282663 1042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:49:04.286185 1056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:04.287552 1042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:04.287729 1042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "ff8536b6644b4891bdb9d6e424b89080"
format_stamp: "Formatted at 2026-06-20 18:49:04 on dist-test-slave-zfp8"
server_key: "9f56692c45881b0f07a7c0c06e8b15a9"
server_key_iv: "83ec3386c32181c59e88d1adef59ba8a"
server_key_version: "encryptionkey@0"
I20260620 18:49:04.287948 1042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:04.308701 1042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:04.309576 1042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:04.309890 1042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:04.310878 1042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:04.312433 1042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:04.312566 1042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:04.312707 1042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:04.312805 1042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:04.346889 1042 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:37645
I20260620 18:49:04.346939 1168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:37645 every 8 connection(s)
I20260620 18:49:04.348343 1042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:49:04.351145 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 1042
I20260620 18:49:04.351372 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:49:04.351890 15771 external_mini_cluster.cc:1428] Setting key b57c43066fa231252d8deaea44a13f83
I20260620 18:49:04.359537 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35963
--builtin_ntp_servers=127.15.102.212:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:49:04.363435 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:04.351124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:38931 (local address 127.15.102.254:35963)
0620 18:49:04.351762 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:04.351769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:04.353300 (+ 1531us) server_negotiation.cc:407] Connection header received
0620 18:49:04.356338 (+ 3038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:04.356342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:04.356421 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:04.356526 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:04.357902 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:04.358759 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:04.360533 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:04.360958 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:04.361101 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:04.362516 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:04.362521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:04.362524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:04.362544 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:04.362590 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:04.362602 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:04.362699 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:04.362930 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:49:04.363029 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":496,"thread_start_us":365,"threads_started":1}
I20260620 18:49:04.364532 1172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:04.351831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35963 (local address 127.15.102.193:38931)
0620 18:49:04.353080 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:04.353136 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:04.355920 (+ 2784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:04.356827 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:04.356840 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:04.356971 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:04.357724 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:04.357752 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:04.358918 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:04.358923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:04.360381 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:04.360394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:04.361198 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:04.361206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:04.361440 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:04.362178 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:04.362180 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:04.362224 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:04.362312 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:04.362701 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:04.362716 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:04.362725 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 18:49:04.362922 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 18:49:04.363421 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":335,"threads_started":1}
I20260620 18:49:04.366205 1169 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35963
I20260620 18:49:04.366534 1169 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:04.367349 1169 heartbeater.cc:511] Master 127.15.102.254:35963 requested a full tablet report, sending...
I20260620 18:49:04.369676 983 ts_manager.cc:194] Registered new tserver with Master: ff8536b6644b4891bdb9d6e424b89080 (127.15.102.193:37645)
I20260620 18:49:04.371961 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.193:38931
W20260620 18:49:04.698900 1173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:04.699180 1173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:04.699406 1173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:04.706730 1173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:04.706830 1173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:04.706938 1173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:49:04.716820 1173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:04.718248 1173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:04.720145 1173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:04.733582 1181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:04.733613 1179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:04.733692 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:04.734861 1173 server_base.cc:1034] running on GCE node
I20260620 18:49:04.735599 1173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:04.736538 1173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:04.737751 1173 hybrid_clock.cc:648] HybridClock initialized: now 1781981344737693 us; error 47 us; skew 500 ppm
I20260620 18:49:04.738124 1173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:04.740517 1173 webserver.cc:460] Webserver started at http://127.15.102.194:43243/ using document root <none> and password file <none>
I20260620 18:49:04.741240 1173 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:04.741329 1173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:04.741640 1173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:04.744663 1173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "7b97a9c1f2e4464fa76d5170240c67d8"
format_stamp: "Formatted at 2026-06-20 18:49:04 on dist-test-slave-zfp8"
server_key: "1a7d430fee37d5009cfa354cdffbaa75"
server_key_iv: "138039293403e8a2477b80abf924169e"
server_key_version: "encryptionkey@0"
I20260620 18:49:04.745491 1173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "7b97a9c1f2e4464fa76d5170240c67d8"
format_stamp: "Formatted at 2026-06-20 18:49:04 on dist-test-slave-zfp8"
server_key: "1a7d430fee37d5009cfa354cdffbaa75"
server_key_iv: "138039293403e8a2477b80abf924169e"
server_key_version: "encryptionkey@0"
I20260620 18:49:04.750830 1173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 18:49:04.754501 1187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:04.756068 1173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:49:04.756258 1173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "7b97a9c1f2e4464fa76d5170240c67d8"
format_stamp: "Formatted at 2026-06-20 18:49:04 on dist-test-slave-zfp8"
server_key: "1a7d430fee37d5009cfa354cdffbaa75"
server_key_iv: "138039293403e8a2477b80abf924169e"
server_key_version: "encryptionkey@0"
I20260620 18:49:04.756510 1173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:04.775143 1173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:04.776201 1173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:04.776628 1173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:04.777704 1173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:04.779400 1173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:04.779520 1173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:04.779645 1173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:04.779735 1173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:04.814179 1173 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:44137
I20260620 18:49:04.814236 1299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:44137 every 8 connection(s)
I20260620 18:49:04.815634 1173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:49:04.816499 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 1173
I20260620 18:49:04.816720 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:49:04.817296 15771 external_mini_cluster.cc:1428] Setting key 30576925c41dff2ab6d01f66f5d1805f
I20260620 18:49:04.820899 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35963
--builtin_ntp_servers=127.15.102.212:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:49:04.829151 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:04.818662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:52981 (local address 127.15.102.254:35963)
0620 18:49:04.818891 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:04.818896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:04.820607 (+ 1711us) server_negotiation.cc:407] Connection header received
0620 18:49:04.822512 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:04.822517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:04.822607 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:04.822733 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:04.823960 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:04.824683 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:04.826271 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:04.826686 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:04.826828 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:04.828300 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:04.828306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:04.828318 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:04.828338 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:04.828370 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:04.828381 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:04.828480 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:04.828715 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:49:04.828822 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:49:04.830240 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:04.819467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35963 (local address 127.15.102.194:52981)
0620 18:49:04.820442 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:04.820496 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:04.822225 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:04.822985 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:04.822994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:04.823092 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:04.823783 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:04.823800 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:04.824848 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:04.824852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:04.826117 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:04.826129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:04.826923 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:04.826931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:04.827161 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:04.827976 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:04.827978 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:04.828019 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:04.828106 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:04.828486 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:04.828499 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:04.828507 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:49:04.828690 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 18:49:04.829117 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":330,"threads_started":1}
I20260620 18:49:04.831835 1300 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35963
I20260620 18:49:04.832177 1300 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:04.833178 1300 heartbeater.cc:511] Master 127.15.102.254:35963 requested a full tablet report, sending...
I20260620 18:49:04.834904 983 ts_manager.cc:194] Registered new tserver with Master: 7b97a9c1f2e4464fa76d5170240c67d8 (127.15.102.194:44137)
I20260620 18:49:04.836202 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.194:52981
W20260620 18:49:05.166343 1303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:05.166756 1303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:05.166851 1303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:05.174146 1303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:05.174269 1303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:05.174378 1303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:49:05.183650 1303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:35963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:05.184932 1303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:05.186685 1303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:05.200382 1308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:05.200451 1311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:05.200428 1309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:05.201716 1303 server_base.cc:1034] running on GCE node
I20260620 18:49:05.202404 1303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:05.203460 1303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:05.204674 1303 hybrid_clock.cc:648] HybridClock initialized: now 1781981345204629 us; error 39 us; skew 500 ppm
I20260620 18:49:05.205013 1303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:05.207444 1303 webserver.cc:460] Webserver started at http://127.15.102.195:43387/ using document root <none> and password file <none>
I20260620 18:49:05.208190 1303 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:05.208280 1303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:05.208638 1303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:05.211730 1303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "a2cc56a005c6494886869abba24e1123"
format_stamp: "Formatted at 2026-06-20 18:49:05 on dist-test-slave-zfp8"
server_key: "b4f58aab9c0b74b2154bc3454effba62"
server_key_iv: "1156e7d009c9d5e38f40f1b5e7c57ca6"
server_key_version: "encryptionkey@0"
I20260620 18:49:05.212546 1303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "a2cc56a005c6494886869abba24e1123"
format_stamp: "Formatted at 2026-06-20 18:49:05 on dist-test-slave-zfp8"
server_key: "b4f58aab9c0b74b2154bc3454effba62"
server_key_iv: "1156e7d009c9d5e38f40f1b5e7c57ca6"
server_key_version: "encryptionkey@0"
I20260620 18:49:05.218201 1303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:49:05.221851 1317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:05.223387 1303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 18:49:05.223572 1303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "a2cc56a005c6494886869abba24e1123"
format_stamp: "Formatted at 2026-06-20 18:49:05 on dist-test-slave-zfp8"
server_key: "b4f58aab9c0b74b2154bc3454effba62"
server_key_iv: "1156e7d009c9d5e38f40f1b5e7c57ca6"
server_key_version: "encryptionkey@0"
I20260620 18:49:05.223809 1303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:05.236775 1303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:05.237687 1303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:05.238102 1303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:05.239327 1303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:05.240818 1303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:05.240904 1303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:05.240983 1303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:05.241034 1303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:05.274978 1303 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:43729
I20260620 18:49:05.275095 1429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:43729 every 8 connection(s)
I20260620 18:49:05.276389 1303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:49:05.279079 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 1303
I20260620 18:49:05.279307 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:49:05.279867 15771 external_mini_cluster.cc:1428] Setting key 9edfa081b6215e983f61e96f64d59048
I20260620 18:49:05.290102 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.279122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:36573 (local address 127.15.102.254:35963)
0620 18:49:05.279395 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.279403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.281354 (+ 1951us) server_negotiation.cc:407] Connection header received
0620 18:49:05.283720 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.283724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.283794 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.283876 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:05.284992 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.285775 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.287545 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.287955 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.288070 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.289378 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:05.289384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:05.289387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:05.289409 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:05.289442 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:05.289454 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:05.289534 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.289739 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.289822 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 18:49:05.291275 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.279925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:35963 (local address 127.15.102.195:36573)
0620 18:49:05.281139 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.281212 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:05.283457 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:05.284087 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:05.284096 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:05.284214 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:05.284819 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.284837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.285950 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.285955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.287411 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.287425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.288163 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.288168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.288373 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.289059 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:05.289062 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:05.289107 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:05.289192 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:05.289546 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:05.289553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:05.289559 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:49:05.289716 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 18:49:05.290028 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":352,"threads_started":1}
I20260620 18:49:05.293025 1430 heartbeater.cc:347] Connected to a master server at 127.15.102.254:35963
I20260620 18:49:05.293363 1430 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:05.294340 1430 heartbeater.cc:511] Master 127.15.102.254:35963 requested a full tablet report, sending...
I20260620 18:49:05.295964 983 ts_manager.cc:194] Registered new tserver with Master: a2cc56a005c6494886869abba24e1123 (127.15.102.195:43729)
I20260620 18:49:05.297086 983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.195:36573
I20260620 18:49:05.308400 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:49:05.319435 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.312932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.15.102.254:35963)
0620 18:49:05.313182 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.313186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.313609 (+ 423us) server_negotiation.cc:407] Connection header received
0620 18:49:05.313785 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.313789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.313873 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.313971 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:05.314625 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.315492 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.316837 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.317347 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.317509 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.318723 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:05.318730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:05.318733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:05.318755 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:05.318793 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:05.318806 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:05.318892 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.319020 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.319108 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 18:49:05.323855 983 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:49:05.331363 983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:49:05.375161 1169 heartbeater.cc:503] Master 127.15.102.254:35963 was elected leader, sending a full tablet report...
I20260620 18:49:05.393321 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.345735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.15.102.195:43729)
0620 18:49:05.357733 (+ 11998us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.357747 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.384987 (+ 27240us) server_negotiation.cc:407] Connection header received
0620 18:49:05.385327 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.385335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.385558 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.385740 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:05.386745 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.388296 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.390468 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.391949 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.392158 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.392397 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.392768 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.392937 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11736,"thread_start_us":11531,"threads_started":1}
I20260620 18:49:05.393374 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.344951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:43729 (local address 127.0.0.1:60490)
0620 18:49:05.384841 (+ 39890us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.384867 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:05.385109 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:05.385957 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:05.385959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:05.385981 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:05.386540 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.386552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.388431 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.388439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.390267 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.390279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.392159 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.392165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.392451 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.392513 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 18:49:05.392707 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 18:49:05.392914 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39742,"mutex_wait_us":67,"thread_start_us":39604,"threads_started":1}
I20260620 18:49:05.394088 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.344687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.0.0.1:50542)
0620 18:49:05.382439 (+ 37752us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.382519 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:05.382835 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:05.385201 (+ 2366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:05.385212 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:05.385248 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:05.386234 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.386247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.388270 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.388274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.390781 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.390799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.393256 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.393264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.393450 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.393494 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 18:49:05.393587 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:49:05.393728 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37360,"mutex_wait_us":62,"thread_start_us":40923,"threads_started":1}
I20260620 18:49:05.394248 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.344918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.15.102.193:37645)
0620 18:49:05.384464 (+ 39546us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.384475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.384515 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:49:05.384645 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.384654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.384948 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.385803 (+ 855us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:05.386433 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.388120 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.390970 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.392730 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.392912 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.393116 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.393650 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.393824 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39376,"thread_start_us":39255,"threads_started":1}
I20260620 18:49:05.396272 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.345444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:44137 (local address 127.0.0.1:39224)
0620 18:49:05.386333 (+ 40889us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.386361 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:05.386600 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:05.387216 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:05.387221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:05.387268 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:05.388025 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.388038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.390563 (+ 2525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.390569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.393387 (+ 2818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.393402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.395617 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.395623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.395763 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.395783 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 18:49:05.395882 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:49:05.396011 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40745,"thread_start_us":37753,"threads_started":1}
I20260620 18:49:05.397064 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.346137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.15.102.194:44137)
0620 18:49:05.381359 (+ 35222us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.381374 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.386480 (+ 5106us) server_negotiation.cc:407] Connection header received
0620 18:49:05.386860 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.386866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.387022 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.387264 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:05.388802 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.390360 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.393611 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.395078 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.395206 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.395436 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.396573 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.396783 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34936,"thread_start_us":35247,"threads_started":1}
I20260620 18:49:05.398514 1104 tablet_service.cc:1467] Processing CreateTablet for tablet af7d38979a4041f0969fa9f5edb95937 (DEFAULT_TABLE table=test-table [id=5604fd2c522b488b8c19eadeece0f74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:05.400041 1104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7d38979a4041f0969fa9f5edb95937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:05.399909 1235 tablet_service.cc:1467] Processing CreateTablet for tablet af7d38979a4041f0969fa9f5edb95937 (DEFAULT_TABLE table=test-table [id=5604fd2c522b488b8c19eadeece0f74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:05.401252 1235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7d38979a4041f0969fa9f5edb95937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:05.401871 1365 tablet_service.cc:1467] Processing CreateTablet for tablet af7d38979a4041f0969fa9f5edb95937 (DEFAULT_TABLE table=test-table [id=5604fd2c522b488b8c19eadeece0f74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:05.403301 1365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7d38979a4041f0969fa9f5edb95937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:05.412601 1448 tablet_bootstrap.cc:492] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Bootstrap starting.
I20260620 18:49:05.415764 1449 tablet_bootstrap.cc:492] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Bootstrap starting.
I20260620 18:49:05.417573 1448 tablet_bootstrap.cc:654] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:05.417917 1450 tablet_bootstrap.cc:492] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Bootstrap starting.
I20260620 18:49:05.419450 1448 log.cc:824] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:05.419646 1449 tablet_bootstrap.cc:654] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:05.420773 1449 log.cc:824] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:05.421347 1450 tablet_bootstrap.cc:654] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:05.422567 1450 log.cc:824] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:05.422977 1448 tablet_bootstrap.cc:492] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: No bootstrap required, opened a new log
I20260620 18:49:05.423375 1448 ts_tablet_manager.cc:1397] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260620 18:49:05.423460 1449 tablet_bootstrap.cc:492] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: No bootstrap required, opened a new log
I20260620 18:49:05.423828 1449 ts_tablet_manager.cc:1397] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260620 18:49:05.427423 1450 tablet_bootstrap.cc:492] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: No bootstrap required, opened a new log
I20260620 18:49:05.427711 1450 ts_tablet_manager.cc:1397] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 18:49:05.431721 1449 raft_consensus.cc:348] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.432121 1449 raft_consensus.cc:374] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:05.432228 1449 raft_consensus.cc:729] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8536b6644b4891bdb9d6e424b89080, State: Initialized, Role: FOLLOWER
I20260620 18:49:05.432898 1449 consensus_queue.cc:260] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.434300 1448 raft_consensus.cc:348] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.434863 1448 raft_consensus.cc:374] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:05.434960 1448 raft_consensus.cc:729] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b97a9c1f2e4464fa76d5170240c67d8, State: Initialized, Role: FOLLOWER
I20260620 18:49:05.435004 1449 ts_tablet_manager.cc:1428] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 18:49:05.435748 1448 consensus_queue.cc:260] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.437795 1300 heartbeater.cc:503] Master 127.15.102.254:35963 was elected leader, sending a full tablet report...
I20260620 18:49:05.438447 1448 ts_tablet_manager.cc:1428] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260620 18:49:05.438845 1450 raft_consensus.cc:348] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.439558 1450 raft_consensus.cc:374] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:05.439671 1450 raft_consensus.cc:729] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cc56a005c6494886869abba24e1123, State: Initialized, Role: FOLLOWER
I20260620 18:49:05.440351 1450 consensus_queue.cc:260] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.441516 1430 heartbeater.cc:503] Master 127.15.102.254:35963 was elected leader, sending a full tablet report...
I20260620 18:49:05.442763 1450 ts_tablet_manager.cc:1428] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260620 18:49:05.528254 1431 tablet.cc:2369] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:05.545190 1455 raft_consensus.cc:482] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:49:05.545538 1455 raft_consensus.cc:504] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.547290 1455 leader_election.cc:288] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8536b6644b4891bdb9d6e424b89080 (127.15.102.193:37645), a2cc56a005c6494886869abba24e1123 (127.15.102.195:43729)
I20260620 18:49:05.556950 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.547773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:41413 (local address 127.15.102.195:43729)
0620 18:49:05.547982 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.547989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.548726 (+ 737us) server_negotiation.cc:407] Connection header received
0620 18:49:05.548925 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.548931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.549015 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.549122 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:05.550079 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.551164 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.554469 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.555769 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.555913 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.555986 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.556554 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.556694 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:49:05.556962 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.547944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:43421 (local address 127.15.102.193:37645)
0620 18:49:05.548152 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.548160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.548725 (+ 565us) server_negotiation.cc:407] Connection header received
0620 18:49:05.548925 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.548931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.549015 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.549136 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:05.550083 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.551153 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.554469 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.555740 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.555913 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.555986 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.556554 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.556694 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:49:05.557353 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.547697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.15.102.194:43421)
0620 18:49:05.548619 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.548635 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:05.548792 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:05.549182 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:05.549187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:05.549235 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:05.549896 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.549923 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.551334 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.551339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.554222 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.554239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.556021 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.556028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.556309 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.556359 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 18:49:05.556471 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 18:49:05.556642 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":398,"spinlock_wait_cycles":48640,"thread_start_us":446,"threads_started":1}
I20260620 18:49:05.557386 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.547561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:43729 (local address 127.15.102.194:41413)
0620 18:49:05.548568 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.548588 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:05.548755 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:05.549182 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:05.549187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:05.549235 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:05.549924 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.549933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.551333 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.551339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.554228 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:05.554239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.556035 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.556042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:05.556304 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.556369 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 18:49:05.556467 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:49:05.556642 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":374,"spinlock_wait_cycles":15872,"thread_start_us":761,"threads_started":1}
I20260620 18:49:05.558548 1385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7d38979a4041f0969fa9f5edb95937" candidate_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cc56a005c6494886869abba24e1123" is_pre_election: true
I20260620 18:49:05.558554 1124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7d38979a4041f0969fa9f5edb95937" candidate_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8536b6644b4891bdb9d6e424b89080" is_pre_election: true
I20260620 18:49:05.559049 1124 raft_consensus.cc:2454] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b97a9c1f2e4464fa76d5170240c67d8 in term 0.
I20260620 18:49:05.559055 1385 raft_consensus.cc:2454] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b97a9c1f2e4464fa76d5170240c67d8 in term 0.
I20260620 18:49:05.559829 1190 leader_election.cc:302] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7b97a9c1f2e4464fa76d5170240c67d8, a2cc56a005c6494886869abba24e1123, ff8536b6644b4891bdb9d6e424b89080; no voters:
I20260620 18:49:05.560220 1455 raft_consensus.cc:2781] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:49:05.560369 1455 raft_consensus.cc:482] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:49:05.560494 1455 raft_consensus.cc:3037] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:05.562855 1455 raft_consensus.cc:504] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.563580 1455 leader_election.cc:288] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [CANDIDATE]: Term 1 election: Requested vote from peers ff8536b6644b4891bdb9d6e424b89080 (127.15.102.193:37645), a2cc56a005c6494886869abba24e1123 (127.15.102.195:43729)
I20260620 18:49:05.564075 1124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7d38979a4041f0969fa9f5edb95937" candidate_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8536b6644b4891bdb9d6e424b89080"
I20260620 18:49:05.564119 1385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7d38979a4041f0969fa9f5edb95937" candidate_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cc56a005c6494886869abba24e1123"
I20260620 18:49:05.564307 1385 raft_consensus.cc:3037] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:05.564307 1124 raft_consensus.cc:3037] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:05.567577 1385 raft_consensus.cc:2454] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b97a9c1f2e4464fa76d5170240c67d8 in term 1.
I20260620 18:49:05.567597 1124 raft_consensus.cc:2454] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b97a9c1f2e4464fa76d5170240c67d8 in term 1.
I20260620 18:49:05.568037 1190 leader_election.cc:302] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b97a9c1f2e4464fa76d5170240c67d8, ff8536b6644b4891bdb9d6e424b89080; no voters:
W20260620 18:49:05.568037 1301 tablet.cc:2369] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:05.568377 1455 raft_consensus.cc:2781] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:05.569097 1455 raft_consensus.cc:686] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 1 LEADER]: Becoming Leader. State: Replica: 7b97a9c1f2e4464fa76d5170240c67d8, State: Running, Role: LEADER
I20260620 18:49:05.569729 1455 consensus_queue.cc:237] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } }
I20260620 18:49:05.575187 982 catalog_manager.cc:5495] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b97a9c1f2e4464fa76d5170240c67d8 (127.15.102.194). New cstate: current_term: 1 leader_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b97a9c1f2e4464fa76d5170240c67d8" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 44137 } health_report { overall_health: HEALTHY } } }
W20260620 18:49:05.600936 1170 tablet.cc:2369] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:05.635545 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.629828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39240 (local address 127.15.102.194:44137)
0620 18:49:05.630050 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.630055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.630074 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:49:05.630254 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.630259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.630357 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.630489 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:05.630841 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.631599 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.633230 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.633679 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.633807 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.634316 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:05.634942 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:05.635063 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.635204 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.635326 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:49:05.644011 1124 raft_consensus.cc:1261] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 1 FOLLOWER]: Refusing update from remote peer 7b97a9c1f2e4464fa76d5170240c67d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:05.644019 1385 raft_consensus.cc:1261] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 1 FOLLOWER]: Refusing update from remote peer 7b97a9c1f2e4464fa76d5170240c67d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:05.645095 1459 consensus_queue.cc:1035] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8536b6644b4891bdb9d6e424b89080" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:05.645447 1455 consensus_queue.cc:1035] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cc56a005c6494886869abba24e1123" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:05.655751 1463 mvcc.cc:205] Tried to move back new op lower bound from 7298995591746932736 to 7298995591458066432. Current Snapshot: MvccSnapshot[applied={T|T < 7298995591746932736}]
I20260620 18:49:05.672966 983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:05.673269 983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:05.677136 983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f13d01e8c97f401da72ea6fe5d6e1bf9: Sending DeleteTablet for 3 replicas of tablet af7d38979a4041f0969fa9f5edb95937
I20260620 18:49:05.678406 1235 tablet_service.cc:1514] Processing DeleteTablet for tablet af7d38979a4041f0969fa9f5edb95937 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:05 UTC) from {username='slave'} at 127.0.0.1:39224
I20260620 18:49:05.678388 1365 tablet_service.cc:1514] Processing DeleteTablet for tablet af7d38979a4041f0969fa9f5edb95937 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:05 UTC) from {username='slave'} at 127.0.0.1:60490
I20260620 18:49:05.678537 1104 tablet_service.cc:1514] Processing DeleteTablet for tablet af7d38979a4041f0969fa9f5edb95937 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:05 UTC) from {username='slave'} at 127.0.0.1:50542
I20260620 18:49:05.680770 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 1042
I20260620 18:49:05.691009 1474 tablet_replica.cc:330] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: stopping tablet replica
I20260620 18:49:05.691733 1474 raft_consensus.cc:2229] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:05.692404 1474 raft_consensus.cc:2258] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:05.694052 1473 tablet_replica.cc:330] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: stopping tablet replica
I20260620 18:49:05.694581 1473 raft_consensus.cc:2229] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:05.695155 1473 raft_consensus.cc:2258] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:05.697856 1473 ts_tablet_manager.cc:1905] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:05.697858 1474 ts_tablet_manager.cc:1905] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:05.698048 1472 tablet_replica.cc:330] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: stopping tablet replica
I20260620 18:49:05.698578 1472 raft_consensus.cc:2229] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:49:05.703176 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.682403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.15.102.193:37645)
0620 18:49:05.682649 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.682659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.693379 (+ 10720us) server_negotiation.cc:407] Connection header received
0620 18:49:05.693630 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.693639 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.693776 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.693913 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:05.694822 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.696572 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.698485 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.699117 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.699226 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.701165 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:05.701177 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:05.701196 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:05.702046 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:05.702140 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:05.702167 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:05.702267 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.702592 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.702732 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:49:05.704406 1164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:05.704388 1472 raft_consensus.cc:2258] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:05.706997 1472 ts_tablet_manager.cc:1905] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:05.708294 1473 ts_tablet_manager.cc:1918] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:05.708503 1473 log.cc:1196] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/af7d38979a4041f0969fa9f5edb95937
I20260620 18:49:05.708988 1473 ts_tablet_manager.cc:1939] T af7d38979a4041f0969fa9f5edb95937 P a2cc56a005c6494886869abba24e1123: Deleting consensus metadata
I20260620 18:49:05.710592 967 catalog_manager.cc:4841] TS a2cc56a005c6494886869abba24e1123 (127.15.102.195:43729): tablet af7d38979a4041f0969fa9f5edb95937 (table test-table [id=5604fd2c522b488b8c19eadeece0f74e]) successfully deleted
I20260620 18:49:05.715157 1472 ts_tablet_manager.cc:1918] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:05.715363 1472 log.cc:1196] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/af7d38979a4041f0969fa9f5edb95937
I20260620 18:49:05.715938 1472 ts_tablet_manager.cc:1939] T af7d38979a4041f0969fa9f5edb95937 P 7b97a9c1f2e4464fa76d5170240c67d8: Deleting consensus metadata
I20260620 18:49:05.717897 970 catalog_manager.cc:4841] TS 7b97a9c1f2e4464fa76d5170240c67d8 (127.15.102.194:44137): tablet af7d38979a4041f0969fa9f5edb95937 (table test-table [id=5604fd2c522b488b8c19eadeece0f74e]) successfully deleted
I20260620 18:49:05.820724 1474 ts_tablet_manager.cc:1918] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:05.820986 1474 log.cc:1196] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/af7d38979a4041f0969fa9f5edb95937
I20260620 18:49:05.821386 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 1042
I20260620 18:49:05.821533 1474 ts_tablet_manager.cc:1939] T af7d38979a4041f0969fa9f5edb95937 P ff8536b6644b4891bdb9d6e424b89080: Deleting consensus metadata
I20260620 18:49:05.845974 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 1173
W20260620 18:49:05.846124 969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:37645 (error 108)
I20260620 18:49:05.848938 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.848497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.0.0.1:50560)
0620 18:49:05.848706 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.848806 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 18:49:05.849220 969 catalog_manager.cc:4584] TS ff8536b6644b4891bdb9d6e424b89080 (127.15.102.193:37645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af7d38979a4041f0969fa9f5edb95937: Network error: Client connection negotiation failed: client connection to 127.15.102.193:37645: connect: Connection refused (error 111)
I20260620 18:49:05.854849 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.846833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.15.102.194:44137)
0620 18:49:05.847059 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.847068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.847299 (+ 231us) server_negotiation.cc:407] Connection header received
0620 18:49:05.847743 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.847749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.847853 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.847976 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:05.848637 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.849894 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.851182 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.852021 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.852159 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.853243 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:05.853254 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:05.853272 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:05.854072 (+ 800us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:05.854140 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:05.854170 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:05.854277 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.854369 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.854477 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:49:05.855659 1295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:05.902719 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.902258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.0.0.1:50568)
0620 18:49:05.902466 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.902571 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 18:49:05.903785 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.903471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.0.0.1:50584)
0620 18:49:05.903608 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.903675 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:49:05.964661 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.964087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.0.0.1:50592)
0620 18:49:05.964366 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.964462 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 18:49:05.965723 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.965412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.0.0.1:50602)
0620 18:49:05.965538 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:05.965603 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 18:49:05.971252 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 1173
I20260620 18:49:05.990734 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 1303
I20260620 18:49:05.997539 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:05.991459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.15.102.195:43729)
0620 18:49:05.991633 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:05.991637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:05.991686 (+ 49us) server_negotiation.cc:407] Connection header received
0620 18:49:05.991853 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:05.991857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:05.991946 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:05.992036 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:05.992405 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.993165 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.994480 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:05.994855 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:05.994946 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:05.996181 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:05.996191 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:05.996206 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:05.996765 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:05.996822 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:05.996842 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:05.996935 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:05.997189 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 18:49:05.997280 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:49:05.998207 1425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:06.046994 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:06.046454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.0.0.1:50612)
0620 18:49:06.046696 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:06.046797 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 18:49:06.048146 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:06.047733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:37645 (local address 127.0.0.1:50616)
0620 18:49:06.047949 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:06.048030 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 18:49:06.087641 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 1303
I20260620 18:49:06.105245 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 952
I20260620 18:49:06.106511 1013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:06.219215 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 952
2026-06-20T18:49:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2959 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:06 dist-test-slave-zfp8 krb5kdc[1483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:06 dist-test-slave-zfp8 krb5kdc[1483](info): set up 2 sockets
Jun 20 18:49:06 dist-test-slave-zfp8 krb5kdc[1483](info): commencing operation
krb5kdc: starting...
W20260620 18:49:07.347445 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:07 dist-test-slave-zfp8 krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:07.596380 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36963
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:36963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:07.932447 1497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:07.932818 1497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:07.933001 1497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:07.940816 1497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:07.940940 1497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:07.941038 1497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:07.941138 1497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:07.941213 1497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:07.951712 1497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:36963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:07.953171 1497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:07.954555 1497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:07.966936 1502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:07.966935 1503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:07.966962 1505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:07.967912 1497 server_base.cc:1034] running on GCE node
I20260620 18:49:07.968806 1497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:07.970440 1497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:07.971686 1497 hybrid_clock.cc:648] HybridClock initialized: now 1781981347971635 us; error 74 us; skew 500 ppm
I20260620 18:49:07.972061 1497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:07 dist-test-slave-zfp8 krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981347, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:07.975327 1497 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:07.976969 1497 webserver.cc:460] Webserver started at http://127.15.102.254:43133/ using document root <none> and password file <none>
I20260620 18:49:07.977720 1497 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:07.977828 1497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:07.978099 1497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:07.981372 1497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "cfc9dc0878b04b5986163c56130b99a1"
format_stamp: "Formatted at 2026-06-20 18:49:07 on dist-test-slave-zfp8"
server_key: "9fec9984a606ecb91ecc74f36e43e2d7"
server_key_iv: "9e76c7499321f5cb0c026d3c9fbf2f83"
server_key_version: "encryptionkey@0"
I20260620 18:49:07.982113 1497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "cfc9dc0878b04b5986163c56130b99a1"
format_stamp: "Formatted at 2026-06-20 18:49:07 on dist-test-slave-zfp8"
server_key: "9fec9984a606ecb91ecc74f36e43e2d7"
server_key_iv: "9e76c7499321f5cb0c026d3c9fbf2f83"
server_key_version: "encryptionkey@0"
I20260620 18:49:07.987708 1497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:49:07.991447 1512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:07.992955 1497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:07.993124 1497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "cfc9dc0878b04b5986163c56130b99a1"
format_stamp: "Formatted at 2026-06-20 18:49:07 on dist-test-slave-zfp8"
server_key: "9fec9984a606ecb91ecc74f36e43e2d7"
server_key_iv: "9e76c7499321f5cb0c026d3c9fbf2f83"
server_key_version: "encryptionkey@0"
I20260620 18:49:07.993304 1497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:08.024372 1497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:08.028657 1497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:08.028904 1497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:08.048261 1497 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:36963
I20260620 18:49:08.048285 1563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:36963 every 8 connection(s)
I20260620 18:49:08.049788 1497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:08.051348 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 1497
I20260620 18:49:08.051541 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:08.052105 15771 external_mini_cluster.cc:1428] Setting key b5c6b3ae8c2cc69334e65ed94469c8fd
I20260620 18:49:08.054778 1564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:49:08 dist-test-slave-zfp8 krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:08.069182 1564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1: Bootstrap starting.
I20260620 18:49:08.072731 1564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:08.073824 1564 log.cc:824] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:08.076860 1567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:08.054094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.15.102.254:36963)
0620 18:49:08.054975 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:08.054990 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:08.055034 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:49:08.056764 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:08.056788 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:08.057200 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:08.057577 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:08.059524 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:08.061082 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:08.063081 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:08.063911 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:08.064057 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:08.069122 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:08.069131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:08.069148 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:08.069194 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:08.072445 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:08.073290 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:08.073297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:08.073304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:08.073478 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:08.073883 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:08.073890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:08.073893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:08.074517 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:08.074879 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:08.075355 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 18:49:08.075841 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":389,"threads_started":1}
I20260620 18:49:08.077204 1564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1: No bootstrap required, opened a new log
I20260620 18:49:08.085469 1564 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc9dc0878b04b5986163c56130b99a1" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36963 } }
I20260620 18:49:08.086045 1564 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:08.086171 1564 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc9dc0878b04b5986163c56130b99a1, State: Initialized, Role: FOLLOWER
I20260620 18:49:08.086851 1564 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc9dc0878b04b5986163c56130b99a1" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36963 } }
I20260620 18:49:08.087129 1564 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:08.087306 1564 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:08.087505 1564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:08.089704 1564 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc9dc0878b04b5986163c56130b99a1" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36963 } }
I20260620 18:49:08.090240 1564 leader_election.cc:302] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfc9dc0878b04b5986163c56130b99a1; no voters:
I20260620 18:49:08.090835 1564 leader_election.cc:288] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:08.090991 1569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:08.091748 1569 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [term 1 LEADER]: Becoming Leader. State: Replica: cfc9dc0878b04b5986163c56130b99a1, State: Running, Role: LEADER
I20260620 18:49:08.092298 1569 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc9dc0878b04b5986163c56130b99a1" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36963 } }
I20260620 18:49:08.093108 1564 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:08.094887 1570 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfc9dc0878b04b5986163c56130b99a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc9dc0878b04b5986163c56130b99a1" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36963 } } }
I20260620 18:49:08.095312 1570 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:08.095304 1571 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfc9dc0878b04b5986163c56130b99a1. Latest consensus state: current_term: 1 leader_uuid: "cfc9dc0878b04b5986163c56130b99a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfc9dc0878b04b5986163c56130b99a1" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36963 } } }
I20260620 18:49:08.096705 1571 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:08.098007 1574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:08.103832 1574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:08.113519 1574 catalog_manager.cc:1347] Generated new cluster ID: f720f0ce21e74a4ca7b0095ce276cbc4
I20260620 18:49:08.113641 1574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:08.140751 1574 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:08.142663 1574 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:08.159786 1574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1: Generated new TSK 0
I20260620 18:49:08.161036 1574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:49:08.218381 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36963
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:08.565182 1590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:08.565493 1590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:08.565629 1590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:08.573180 1590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:08.573338 1590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:08.573542 1590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:49:08.584757 1590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:08.586158 1590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:08.587641 1590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:08.601717 1595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:08.601795 1598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:08.601754 1596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:08.602788 1590 server_base.cc:1034] running on GCE node
I20260620 18:49:08.603610 1590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:08.604641 1590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:08.605882 1590 hybrid_clock.cc:648] HybridClock initialized: now 1781981348605811 us; error 67 us; skew 500 ppm
I20260620 18:49:08.606225 1590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:08 dist-test-slave-zfp8 krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981348, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:08.609584 1590 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:49:08.611169 1590 webserver.cc:460] Webserver started at http://127.15.102.193:43359/ using document root <none> and password file <none>
I20260620 18:49:08.611958 1590 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:08.612111 1590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:08.612460 1590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:08.615568 1590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "f3600a04dd454e72bf6207bda54af382"
format_stamp: "Formatted at 2026-06-20 18:49:08 on dist-test-slave-zfp8"
server_key: "3d36d28837bef82d782456f4b995affd"
server_key_iv: "17abf97869d0e594c05bcfa45e3a3110"
server_key_version: "encryptionkey@0"
I20260620 18:49:08.616366 1590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "f3600a04dd454e72bf6207bda54af382"
format_stamp: "Formatted at 2026-06-20 18:49:08 on dist-test-slave-zfp8"
server_key: "3d36d28837bef82d782456f4b995affd"
server_key_iv: "17abf97869d0e594c05bcfa45e3a3110"
server_key_version: "encryptionkey@0"
I20260620 18:49:08.621781 1590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:49:08.625687 1605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:08.627162 1590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:49:08.627373 1590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "f3600a04dd454e72bf6207bda54af382"
format_stamp: "Formatted at 2026-06-20 18:49:08 on dist-test-slave-zfp8"
server_key: "3d36d28837bef82d782456f4b995affd"
server_key_iv: "17abf97869d0e594c05bcfa45e3a3110"
server_key_version: "encryptionkey@0"
I20260620 18:49:08.627596 1590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:08.650389 1590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:08.655323 1590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:08.655639 1590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:08.656678 1590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:08.658155 1590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:08.658265 1590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:08.658419 1590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:08.658492 1590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:08.692760 1590 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:39671
I20260620 18:49:08.692866 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:39671 every 8 connection(s)
I20260620 18:49:08.694245 1590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:49:08.704233 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 1590
I20260620 18:49:08.704466 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:49:08.704931 15771 external_mini_cluster.cc:1428] Setting key 171cf8a21d94d207520e7cde93bf85d7
Jun 20 18:49:08 dist-test-slave-zfp8 krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981348, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:08.717062 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:08.697024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:57525 (local address 127.15.102.254:36963)
0620 18:49:08.697538 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:08.697543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:08.698746 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 18:49:08.700789 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:08.700795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:08.700888 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:08.701019 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:08.702626 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:08.703435 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:08.705205 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:08.705614 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:08.705782 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:08.709901 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:08.709910 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:08.709917 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:08.709966 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:08.713672 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:08.714503 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:08.714513 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:08.714519 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:08.714684 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:08.715121 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:08.715129 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:08.715135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:08.715569 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:08.715743 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:08.716597 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 18:49:08.716785 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":228,"threads_started":1}
I20260620 18:49:08.718673 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:08.697744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36963 (local address 127.15.102.193:57525)
0620 18:49:08.698561 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:08.698607 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:08.700498 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:08.701280 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:08.701289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:08.701838 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:08.702437 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:08.702456 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:08.703610 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:08.703616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:08.705011 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:08.705053 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:08.705871 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:08.705876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:08.706184 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:08.707005 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:08.707037 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:08.709641 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:08.713852 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:08.713860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:08.713875 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:08.714361 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:08.714798 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:08.714801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:08.714803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:08.715003 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:08.715705 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:08.715711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:08.716130 (+ 419us) client_negotiation.cc:769] Sending connection context
0620 18:49:08.716497 (+ 367us) client_negotiation.cc:240] Negotiation successful
0620 18:49:08.717131 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":300,"threads_started":1}
I20260620 18:49:08.720444 1718 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36963
I20260620 18:49:08.720786 1718 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:08.721561 1718 heartbeater.cc:511] Master 127.15.102.254:36963 requested a full tablet report, sending...
I20260620 18:49:08.723896 1529 ts_manager.cc:194] Registered new tserver with Master: f3600a04dd454e72bf6207bda54af382 (127.15.102.193:39671)
I20260620 18:49:08.725382 1529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:57525
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:49:08.763384 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36963
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:09.101991 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:09.102358 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:09.102530 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:09.110005 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:09.110195 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:09.110386 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:49:09.120725 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:09.122162 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:09.123569 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:09.137619 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:09.137583 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:09.137671 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:09.138458 1724 server_base.cc:1034] running on GCE node
I20260620 18:49:09.139113 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:09.140211 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:09.141469 1724 hybrid_clock.cc:648] HybridClock initialized: now 1781981349141425 us; error 36 us; skew 500 ppm
I20260620 18:49:09.141815 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:09 dist-test-slave-zfp8 krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981349, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:09.148798 1724 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:49:09.150449 1724 webserver.cc:460] Webserver started at http://127.15.102.194:33825/ using document root <none> and password file <none>
I20260620 18:49:09.151206 1724 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:09.151418 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:09.151757 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:09.154824 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "f9c16e1fd2ae4f59b3880dee29173baf"
format_stamp: "Formatted at 2026-06-20 18:49:09 on dist-test-slave-zfp8"
server_key: "dcd9599ae10a04fe35c15ef8a3df0200"
server_key_iv: "50fa3af7fa0618707ea2f315f6be187c"
server_key_version: "encryptionkey@0"
I20260620 18:49:09.155681 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "f9c16e1fd2ae4f59b3880dee29173baf"
format_stamp: "Formatted at 2026-06-20 18:49:09 on dist-test-slave-zfp8"
server_key: "dcd9599ae10a04fe35c15ef8a3df0200"
server_key_iv: "50fa3af7fa0618707ea2f315f6be187c"
server_key_version: "encryptionkey@0"
I20260620 18:49:09.161361 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:49:09.165060 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:09.166510 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:49:09.166706 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "f9c16e1fd2ae4f59b3880dee29173baf"
format_stamp: "Formatted at 2026-06-20 18:49:09 on dist-test-slave-zfp8"
server_key: "dcd9599ae10a04fe35c15ef8a3df0200"
server_key_iv: "50fa3af7fa0618707ea2f315f6be187c"
server_key_version: "encryptionkey@0"
I20260620 18:49:09.166945 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:09.193084 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:09.198124 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:09.198422 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:09.199542 1724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:09.200927 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:09.201002 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:09.201184 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:09.201234 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:09.234308 1724 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:39843
I20260620 18:49:09.234354 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:39843 every 8 connection(s)
I20260620 18:49:09.235730 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:49:09.239043 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 1724
I20260620 18:49:09.239286 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:49:09.239897 15771 external_mini_cluster.cc:1428] Setting key f6f373b0cb202ed41feb74d289f5282a
Jun 20 18:49:09 dist-test-slave-zfp8 krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981349, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:09.261428 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.238403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:34305 (local address 127.15.102.254:36963)
0620 18:49:09.238888 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:09.238895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:09.240259 (+ 1364us) server_negotiation.cc:407] Connection header received
0620 18:49:09.242421 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:09.242425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:09.242543 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:09.242676 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:09.244960 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.245705 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.248354 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.248801 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.248905 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.253571 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:09.253581 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:09.253590 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:09.253641 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:09.257393 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:09.258315 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:09.258324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:09.258330 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:09.258496 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:09.259107 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:09.259115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:09.259121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:09.259588 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:09.259742 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:09.260651 (+ 909us) server_negotiation.cc:299] Negotiation successful
0620 18:49:09.260922 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":205,"threads_started":1}
I20260620 18:49:09.262216 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.239075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36963 (local address 127.15.102.194:34305)
0620 18:49:09.240046 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:09.240111 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:09.242102 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:09.242940 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:09.242955 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:09.243852 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:09.244757 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.244787 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.245882 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.245890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.248159 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.248187 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.249065 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.249075 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.249421 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.250225 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:09.250260 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:09.253250 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:09.257601 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:09.257610 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:09.257625 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:09.258124 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:09.258707 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:09.258714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:09.258717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:09.258928 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:09.259734 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:09.259741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:09.260116 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 18:49:09.260438 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 18:49:09.260884 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":385,"threads_started":1}
I20260620 18:49:09.263881 1852 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36963
I20260620 18:49:09.264250 1852 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:09.265080 1852 heartbeater.cc:511] Master 127.15.102.254:36963 requested a full tablet report, sending...
I20260620 18:49:09.266721 1529 ts_manager.cc:194] Registered new tserver with Master: f9c16e1fd2ae4f59b3880dee29173baf (127.15.102.194:39843)
I20260620 18:49:09.268043 1529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:34305
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:49:09.300374 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36963
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:09.640623 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:09.640924 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:09.641093 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:09.648667 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:09.648793 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:09.648988 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:49:09.659682 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:09.661069 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:09.662487 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:09.676468 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:09.676513 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:09.676476 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:09.677414 1858 server_base.cc:1034] running on GCE node
I20260620 18:49:09.678068 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:09.679018 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:09.680225 1858 hybrid_clock.cc:648] HybridClock initialized: now 1781981349680158 us; error 57 us; skew 500 ppm
I20260620 18:49:09.680536 1858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:09 dist-test-slave-zfp8 krb5kdc[1483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981349, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:09.683727 1858 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:49:09.685238 1858 webserver.cc:460] Webserver started at http://127.15.102.195:34779/ using document root <none> and password file <none>
I20260620 18:49:09.685981 1858 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:09.686136 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:09.686480 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:09.689551 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "b7990b0e0faa4ee7817ee159bff95488"
format_stamp: "Formatted at 2026-06-20 18:49:09 on dist-test-slave-zfp8"
server_key: "86fe352df07e14d33af8d872208cf070"
server_key_iv: "8f8400e65a192dfca9acadb295129ad9"
server_key_version: "encryptionkey@0"
I20260620 18:49:09.690358 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "b7990b0e0faa4ee7817ee159bff95488"
format_stamp: "Formatted at 2026-06-20 18:49:09 on dist-test-slave-zfp8"
server_key: "86fe352df07e14d33af8d872208cf070"
server_key_iv: "8f8400e65a192dfca9acadb295129ad9"
server_key_version: "encryptionkey@0"
I20260620 18:49:09.696475 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 18:49:09.700513 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:09.701937 1858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:09.702136 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "b7990b0e0faa4ee7817ee159bff95488"
format_stamp: "Formatted at 2026-06-20 18:49:09 on dist-test-slave-zfp8"
server_key: "86fe352df07e14d33af8d872208cf070"
server_key_iv: "8f8400e65a192dfca9acadb295129ad9"
server_key_version: "encryptionkey@0"
I20260620 18:49:09.702353 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:09.728755 1718 heartbeater.cc:503] Master 127.15.102.254:36963 was elected leader, sending a full tablet report...
I20260620 18:49:09.744973 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:09.749143 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:09.749460 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:09.750453 1858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:09.751928 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:09.752036 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:09.752197 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:09.752270 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:09.784421 1858 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:44447
I20260620 18:49:09.784524 1985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:44447 every 8 connection(s)
I20260620 18:49:09.785987 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:49:09.787020 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 1858
I20260620 18:49:09.787233 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:49:09.787714 15771 external_mini_cluster.cc:1428] Setting key acd41f07da543ef910d2f2580aa6da5a
Jun 20 18:49:09 dist-test-slave-zfp8 krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981349, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:09.808542 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.789056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:59033 (local address 127.15.102.254:36963)
0620 18:49:09.789533 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:09.789539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:09.791067 (+ 1528us) server_negotiation.cc:407] Connection header received
0620 18:49:09.793094 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:09.793099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:09.793180 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:09.793269 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:09.794938 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.795782 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.797625 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.798206 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.798329 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.802277 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:09.802285 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:09.802289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:09.802348 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:09.805411 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:09.806189 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:09.806194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:09.806217 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:09.806365 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:09.806835 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:09.806839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:09.806841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:09.807180 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:09.807374 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:09.808074 (+ 700us) server_negotiation.cc:299] Negotiation successful
0620 18:49:09.808256 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":213,"threads_started":1}
I20260620 18:49:09.809609 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.789891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36963 (local address 127.15.102.195:59033)
0620 18:49:09.790864 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:09.790936 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:09.792791 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:09.793544 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:09.793552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:09.794079 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:09.794758 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.794778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.795961 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.795965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.797480 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.797495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.798451 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.798457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.798676 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.799480 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:09.799508 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:09.801960 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:09.805574 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:09.805581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:09.805597 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:09.806069 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:09.806495 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:09.806499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:09.806501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:09.806710 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:09.807365 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:09.807372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:09.807731 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 18:49:09.807978 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 18:49:09.808416 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":403,"threads_started":1}
I20260620 18:49:09.811022 1986 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36963
I20260620 18:49:09.811403 1986 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:09.812183 1986 heartbeater.cc:511] Master 127.15.102.254:36963 requested a full tablet report, sending...
I20260620 18:49:09.813843 1529 ts_manager.cc:194] Registered new tserver with Master: b7990b0e0faa4ee7817ee159bff95488 (127.15.102.195:44447)
I20260620 18:49:09.814733 1529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:59033
I20260620 18:49:09.816792 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:49:09.835058 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.821053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.15.102.254:36963)
0620 18:49:09.821280 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:09.821289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:09.821685 (+ 396us) server_negotiation.cc:407] Connection header received
0620 18:49:09.821903 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:09.821906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:09.821974 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:09.822086 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:09.823586 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.824337 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.825764 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.826165 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.826308 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.829213 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:09.829218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:09.829221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:09.829258 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:09.831874 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:09.832857 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:09.832863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:09.832869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:09.833008 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:09.833493 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:09.833502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:09.833504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:09.833840 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:09.834131 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:09.834551 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 18:49:09.834766 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:49:09.839409 1528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:49:09.847218 1528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:49:09.873697 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.860131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:44447 (local address 127.0.0.1:51800)
0620 18:49:09.863033 (+ 2902us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:09.863053 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:09.863184 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:09.864261 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:09.864269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:09.864309 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:09.864864 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.864879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.867447 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.867456 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.870764 (+ 3308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.870784 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.872480 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.872486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.872718 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.872801 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 18:49:09.872941 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 18:49:09.873157 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2747,"mutex_wait_us":64,"thread_start_us":1698,"threads_started":1}
I20260620 18:49:09.874364 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.860229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.15.102.193:39671)
0620 18:49:09.861769 (+ 1540us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:09.861780 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:09.862750 (+ 970us) server_negotiation.cc:407] Connection header received
0620 18:49:09.862871 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:09.862882 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:09.863151 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:09.863399 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:09.864673 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.866799 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.871361 (+ 4562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.872859 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.873052 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.873240 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:09.873848 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 18:49:09.874024 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1353,"thread_start_us":1202,"threads_started":1}
I20260620 18:49:09.874526 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.860784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.15.102.195:44447)
0620 18:49:09.862276 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:09.862285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:09.863341 (+ 1056us) server_negotiation.cc:407] Connection header received
0620 18:49:09.863458 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:09.863466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:09.863695 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:09.863917 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:09.865108 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.867194 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.870946 (+ 3752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.872274 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.872480 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.872687 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:09.873953 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0620 18:49:09.874133 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1138,"threads_started":1}
I20260620 18:49:09.875365 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.860064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:44422)
0620 18:49:09.862253 (+ 2189us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:09.862305 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:09.862507 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:09.863732 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:09.863744 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:09.863789 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:09.864383 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.864400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.867026 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.867036 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.871162 (+ 4126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.871178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.873409 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.873416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.873611 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.873650 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 18:49:09.874748 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0620 18:49:09.874907 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"mutex_wait_us":1439,"thread_start_us":2644,"threads_started":1}
I20260620 18:49:09.877723 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.863104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:39843 (local address 127.0.0.1:60028)
0620 18:49:09.869271 (+ 6167us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:09.869292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:09.869433 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:09.870344 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:09.870349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:09.870376 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:09.870943 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.870954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.873003 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.873011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.875147 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.875159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.876952 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.876958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.877124 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.877164 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 18:49:09.877273 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:49:09.877402 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6032,"thread_start_us":5866,"threads_started":1}
I20260620 18:49:09.877830 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.863404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.15.102.194:39843)
0620 18:49:09.868701 (+ 5297us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:09.868714 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:09.869417 (+ 703us) server_negotiation.cc:407] Connection header received
0620 18:49:09.869608 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:09.869621 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:09.870146 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:09.870356 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:09.871188 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.872811 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.875327 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.876733 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.876925 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.877092 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:09.877318 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 18:49:09.877503 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5048,"thread_start_us":4905,"threads_started":1}
I20260620 18:49:09.879151 1653 tablet_service.cc:1467] Processing CreateTablet for tablet 8daba537dadb49bda04ab9db083b9527 (DEFAULT_TABLE table=test-table [id=b331af2949d7442685a82c5543e4c106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:09.879114 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 8daba537dadb49bda04ab9db083b9527 (DEFAULT_TABLE table=test-table [id=b331af2949d7442685a82c5543e4c106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:09.880800 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8daba537dadb49bda04ab9db083b9527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:09.880800 1653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8daba537dadb49bda04ab9db083b9527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:09.886467 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 8daba537dadb49bda04ab9db083b9527 (DEFAULT_TABLE table=test-table [id=b331af2949d7442685a82c5543e4c106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:09.887768 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8daba537dadb49bda04ab9db083b9527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:09.896459 2006 tablet_bootstrap.cc:492] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: Bootstrap starting.
I20260620 18:49:09.897300 2005 tablet_bootstrap.cc:492] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Bootstrap starting.
I20260620 18:49:09.900990 2006 tablet_bootstrap.cc:654] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:09.901624 2005 tablet_bootstrap.cc:654] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:09.902150 2006 log.cc:824] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:09.903193 2005 log.cc:824] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:09.904366 2007 tablet_bootstrap.cc:492] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Bootstrap starting.
I20260620 18:49:09.905426 2006 tablet_bootstrap.cc:492] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: No bootstrap required, opened a new log
I20260620 18:49:09.905716 2006 ts_tablet_manager.cc:1397] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 18:49:09.906416 2005 tablet_bootstrap.cc:492] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: No bootstrap required, opened a new log
I20260620 18:49:09.906771 2005 ts_tablet_manager.cc:1397] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 18:49:09.908651 2007 tablet_bootstrap.cc:654] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:09.909960 2007 log.cc:824] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:09.913436 2007 tablet_bootstrap.cc:492] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: No bootstrap required, opened a new log
I20260620 18:49:09.913744 2007 ts_tablet_manager.cc:1397] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 18:49:09.914361 2006 raft_consensus.cc:348] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.914760 2006 raft_consensus.cc:374] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:09.914868 2006 raft_consensus.cc:729] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7990b0e0faa4ee7817ee159bff95488, State: Initialized, Role: FOLLOWER
I20260620 18:49:09.915565 2006 consensus_queue.cc:260] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.917177 2005 raft_consensus.cc:348] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.917867 1986 heartbeater.cc:503] Master 127.15.102.254:36963 was elected leader, sending a full tablet report...
I20260620 18:49:09.918509 2006 ts_tablet_manager.cc:1428] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260620 18:49:09.918943 2005 raft_consensus.cc:374] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:09.919099 2005 raft_consensus.cc:729] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3600a04dd454e72bf6207bda54af382, State: Initialized, Role: FOLLOWER
I20260620 18:49:09.919818 2005 consensus_queue.cc:260] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.921751 2005 ts_tablet_manager.cc:1428] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 18:49:09.925179 2007 raft_consensus.cc:348] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.925675 2007 raft_consensus.cc:374] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:09.925797 2007 raft_consensus.cc:729] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c16e1fd2ae4f59b3880dee29173baf, State: Initialized, Role: FOLLOWER
I20260620 18:49:09.926375 2007 consensus_queue.cc:260] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.927400 1852 heartbeater.cc:503] Master 127.15.102.254:36963 was elected leader, sending a full tablet report...
I20260620 18:49:09.928133 2007 ts_tablet_manager.cc:1428] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
W20260620 18:49:09.946769 1719 tablet.cc:2369] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:09.954049 2011 raft_consensus.cc:482] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:49:09.954388 2011 raft_consensus.cc:504] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.956243 2011 leader_election.cc:288] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3600a04dd454e72bf6207bda54af382 (127.15.102.193:39671), f9c16e1fd2ae4f59b3880dee29173baf (127.15.102.194:39843)
I20260620 18:49:09.962829 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.956587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:44183 (local address 127.15.102.193:39671)
0620 18:49:09.957158 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:09.957166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:09.957192 (+ 26us) server_negotiation.cc:407] Connection header received
0620 18:49:09.957276 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:09.957280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:09.957337 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:09.957443 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:09.957897 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.958859 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.960763 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.962001 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.962133 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.962192 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:09.962538 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 18:49:09.962638 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412}
I20260620 18:49:09.962944 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.956567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.15.102.195:44183)
0620 18:49:09.956975 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:09.956998 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:09.957129 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:09.957467 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:09.957472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:09.957524 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:09.957700 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.957708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.959010 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.959026 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.960520 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.960533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.962159 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.962167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.962362 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.962400 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:49:09.962505 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 18:49:09.962662 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260620 18:49:09.964119 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.956684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:39843 (local address 127.15.102.195:59815)
0620 18:49:09.957674 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:09.957696 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:09.957827 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:09.958347 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:09.958351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:09.958377 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:09.958762 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.958777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.959891 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.959896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.961850 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:09.961866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.963491 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.963500 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:09.963672 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.963697 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 18:49:09.963775 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 18:49:09.963882 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":1091,"threads_started":1}
I20260620 18:49:09.964257 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:09.956916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:59815 (local address 127.15.102.194:39843)
0620 18:49:09.957138 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:09.957144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:09.957806 (+ 662us) server_negotiation.cc:407] Connection header received
0620 18:49:09.958036 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:09.958041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:09.958103 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:09.958152 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:09.958951 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.959727 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.962033 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:09.963305 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:09.963420 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:09.963471 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:09.963844 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 18:49:09.963983 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:49:09.964181 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8daba537dadb49bda04ab9db083b9527" candidate_uuid: "b7990b0e0faa4ee7817ee159bff95488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3600a04dd454e72bf6207bda54af382" is_pre_election: true
I20260620 18:49:09.964843 1673 raft_consensus.cc:2454] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7990b0e0faa4ee7817ee159bff95488 in term 0.
I20260620 18:49:09.965348 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8daba537dadb49bda04ab9db083b9527" candidate_uuid: "b7990b0e0faa4ee7817ee159bff95488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" is_pre_election: true
I20260620 18:49:09.965664 1876 leader_election.cc:302] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7990b0e0faa4ee7817ee159bff95488, f3600a04dd454e72bf6207bda54af382; no voters:
I20260620 18:49:09.965911 1807 raft_consensus.cc:2454] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7990b0e0faa4ee7817ee159bff95488 in term 0.
I20260620 18:49:09.966095 2011 raft_consensus.cc:2781] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:49:09.966249 2011 raft_consensus.cc:482] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:49:09.966382 2011 raft_consensus.cc:3037] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:09.968765 2011 raft_consensus.cc:504] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.969465 2011 leader_election.cc:288] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [CANDIDATE]: Term 1 election: Requested vote from peers f3600a04dd454e72bf6207bda54af382 (127.15.102.193:39671), f9c16e1fd2ae4f59b3880dee29173baf (127.15.102.194:39843)
I20260620 18:49:09.969933 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8daba537dadb49bda04ab9db083b9527" candidate_uuid: "b7990b0e0faa4ee7817ee159bff95488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3600a04dd454e72bf6207bda54af382"
I20260620 18:49:09.970036 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8daba537dadb49bda04ab9db083b9527" candidate_uuid: "b7990b0e0faa4ee7817ee159bff95488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c16e1fd2ae4f59b3880dee29173baf"
I20260620 18:49:09.970178 1673 raft_consensus.cc:3037] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:09.970228 1807 raft_consensus.cc:3037] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:09.973572 1807 raft_consensus.cc:2454] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7990b0e0faa4ee7817ee159bff95488 in term 1.
I20260620 18:49:09.973577 1673 raft_consensus.cc:2454] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7990b0e0faa4ee7817ee159bff95488 in term 1.
I20260620 18:49:09.974089 1876 leader_election.cc:302] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7990b0e0faa4ee7817ee159bff95488, f3600a04dd454e72bf6207bda54af382; no voters:
I20260620 18:49:09.974476 2011 raft_consensus.cc:2781] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:09.974968 2011 raft_consensus.cc:686] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 1 LEADER]: Becoming Leader. State: Replica: b7990b0e0faa4ee7817ee159bff95488, State: Running, Role: LEADER
I20260620 18:49:09.975584 2011 consensus_queue.cc:237] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } }
I20260620 18:49:09.980731 1528 catalog_manager.cc:5495] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7990b0e0faa4ee7817ee159bff95488 (127.15.102.195). New cstate: current_term: 1 leader_uuid: "b7990b0e0faa4ee7817ee159bff95488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7990b0e0faa4ee7817ee159bff95488" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 44447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 } health_report { overall_health: UNKNOWN } } }
W20260620 18:49:09.987808 1853 tablet.cc:2369] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:10.014890 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.009673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.15.102.195:44447)
0620 18:49:10.009861 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:10.009868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:10.009932 (+ 64us) server_negotiation.cc:407] Connection header received
0620 18:49:10.010115 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:10.010118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:10.010206 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:10.010345 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:10.010698 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:10.011484 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:10.012844 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:10.013257 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:10.013358 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:10.013861 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:10.014365 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:10.014498 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:10.014640 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 18:49:10.014681 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:49:10.023710 1807 raft_consensus.cc:1261] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 1 FOLLOWER]: Refusing update from remote peer b7990b0e0faa4ee7817ee159bff95488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:10.023710 1673 raft_consensus.cc:1261] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 1 FOLLOWER]: Refusing update from remote peer b7990b0e0faa4ee7817ee159bff95488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:10.024717 2015 consensus_queue.cc:1035] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3600a04dd454e72bf6207bda54af382" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:10.025151 2011 consensus_queue.cc:1035] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c16e1fd2ae4f59b3880dee29173baf" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:10.038005 2018 mvcc.cc:205] Tried to move back new op lower bound from 7298995609686728704 to 7298995609504063488. Current Snapshot: MvccSnapshot[applied={T|T < 7298995609686728704}]
I20260620 18:49:10.055617 1528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:10.055910 1528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:10.060096 1528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfc9dc0878b04b5986163c56130b99a1: Sending DeleteTablet for 3 replicas of tablet 8daba537dadb49bda04ab9db083b9527
I20260620 18:49:10.061461 1653 tablet_service.cc:1514] Processing DeleteTablet for tablet 8daba537dadb49bda04ab9db083b9527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:10 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:44422
I20260620 18:49:10.061652 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 8daba537dadb49bda04ab9db083b9527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:10 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:51800
I20260620 18:49:10.062224 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 8daba537dadb49bda04ab9db083b9527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:10 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:60028
I20260620 18:49:10.064908 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 1590
I20260620 18:49:10.080470 2028 tablet_replica.cc:330] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: stopping tablet replica
I20260620 18:49:10.081380 2028 raft_consensus.cc:2229] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 1 FOLLOWER]: Raft consensus shutting down.
Jun 20 18:49:10 dist-test-slave-zfp8 krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:49:10.082247 2030 tablet_replica.cc:330] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: stopping tablet replica
I20260620 18:49:10.082640 2030 raft_consensus.cc:2229] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:10.083153 2030 raft_consensus.cc:2258] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:10.083639 2028 raft_consensus.cc:2258] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:10.085884 2028 ts_tablet_manager.cc:1905] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:10.089247 2030 ts_tablet_manager.cc:1905] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:10.093289 2029 tablet_replica.cc:330] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: stopping tablet replica
I20260620 18:49:10.093505 2028 ts_tablet_manager.cc:1918] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:10.093752 2028 log.cc:1196] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-0/wal/wals/8daba537dadb49bda04ab9db083b9527
I20260620 18:49:10.094203 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.066256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.15.102.193:39671)
0620 18:49:10.066484 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:10.066494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:10.067092 (+ 598us) server_negotiation.cc:407] Connection header received
0620 18:49:10.067485 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:10.067494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:10.067616 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:10.067709 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:10.070197 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:10.071830 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:10.075277 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:10.076302 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:10.076458 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:10.084017 (+ 7559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:10.084041 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:10.084068 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:10.084157 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:10.089491 (+ 5334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:10.090867 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:10.090879 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:10.090885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:10.091137 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:10.091790 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:10.091798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:10.091803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:10.092416 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:10.092809 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:10.093497 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 18:49:10.093762 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:49:10.094312 2029 raft_consensus.cc:2229] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:49:10.094484 2028 ts_tablet_manager.cc:1939] T 8daba537dadb49bda04ab9db083b9527 P f3600a04dd454e72bf6207bda54af382: Deleting consensus metadata
I20260620 18:49:10.095005 1713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:10.096576 2029 raft_consensus.cc:2258] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:10.100397 2029 ts_tablet_manager.cc:1905] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:10.102461 2030 ts_tablet_manager.cc:1918] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:10.102650 2030 log.cc:1196] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/8daba537dadb49bda04ab9db083b9527
I20260620 18:49:10.103209 2030 ts_tablet_manager.cc:1939] T 8daba537dadb49bda04ab9db083b9527 P f9c16e1fd2ae4f59b3880dee29173baf: Deleting consensus metadata
I20260620 18:49:10.105355 1513 catalog_manager.cc:4841] TS f9c16e1fd2ae4f59b3880dee29173baf (127.15.102.194:39843): tablet 8daba537dadb49bda04ab9db083b9527 (table test-table [id=b331af2949d7442685a82c5543e4c106]) successfully deleted
I20260620 18:49:10.107460 2029 ts_tablet_manager.cc:1918] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:10.107667 2029 log.cc:1196] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/8daba537dadb49bda04ab9db083b9527
I20260620 18:49:10.108175 2029 ts_tablet_manager.cc:1939] T 8daba537dadb49bda04ab9db083b9527 P b7990b0e0faa4ee7817ee159bff95488: Deleting consensus metadata
I20260620 18:49:10.109750 1514 catalog_manager.cc:4841] TS b7990b0e0faa4ee7817ee159bff95488 (127.15.102.195:44447): tablet 8daba537dadb49bda04ab9db083b9527 (table test-table [id=b331af2949d7442685a82c5543e4c106]) successfully deleted
I20260620 18:49:10.246979 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 1590
W20260620 18:49:10.265321 1515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:39671 (error 108)
I20260620 18:49:10.265471 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 1724
I20260620 18:49:10.267458 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.266958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:44442)
0620 18:49:10.267159 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:10.267314 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 18:49:10.267795 1515 catalog_manager.cc:4584] TS f3600a04dd454e72bf6207bda54af382 (127.15.102.193:39671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8daba537dadb49bda04ab9db083b9527: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Jun 20 18:49:10 dist-test-slave-zfp8 krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:49:10.283980 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.266624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.15.102.194:39843)
0620 18:49:10.266796 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:10.266801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:10.266818 (+ 17us) server_negotiation.cc:407] Connection header received
0620 18:49:10.266875 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:10.266879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:10.266951 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:10.267090 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:10.268871 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:10.269631 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:10.271636 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:10.272106 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:10.272211 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:10.277783 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:10.277800 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:10.277817 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:10.277863 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:10.280906 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:10.281871 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:10.281879 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:10.281881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:10.282013 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:10.282450 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:10.282453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:10.282455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:10.282830 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:10.283076 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:10.283497 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 18:49:10.283673 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:49:10.284672 1847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:10.321532 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.321043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:44458)
0620 18:49:10.321253 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:10.321369 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 18:49:10.322672 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.322312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:44460)
0620 18:49:10.322465 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:10.322546 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 18:49:10.383401 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.382767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:44468)
0620 18:49:10.383034 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:10.383146 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 18:49:10.384632 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.384209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:44482)
0620 18:49:10.384396 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:10.384486 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:49:10.415868 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 1724
I20260620 18:49:10.433367 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 1858
Jun 20 18:49:10 dist-test-slave-zfp8 krb5kdc[1483](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:49:10.450109 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.434193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49428 (local address 127.15.102.195:44447)
0620 18:49:10.434420 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:10.434427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:10.434445 (+ 18us) server_negotiation.cc:407] Connection header received
0620 18:49:10.434660 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:10.434665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:10.434741 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:10.434856 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:10.436275 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:10.437110 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:10.438431 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:10.438901 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:10.439095 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:10.443540 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:10.443553 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:10.443569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:10.443604 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:10.446886 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:10.447824 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:10.447833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:10.447839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:10.448039 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:10.448463 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:10.448470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:10.448472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:10.448857 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:10.449183 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:10.449501 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 18:49:10.449726 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 18:49:10.450886 1981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:10.465307 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.464783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:45012)
0620 18:49:10.465030 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:10.465128 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 18:49:10.466377 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.465978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:45022)
0620 18:49:10.466179 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:10.466260 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 18:49:10.576198 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 1858
I20260620 18:49:10.595896 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 1497
I20260620 18:49:10.597209 1559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:10.710902 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:10.710331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:39671 (local address 127.0.0.1:45026)
0620 18:49:10.710596 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:10.710718 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 18:49:10.711109 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 1497
2026-06-20T18:49:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4490 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T18:49:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:10Z Disabled control of system clock
I20260620 18:49:10.787990 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36485
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:36485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:49:11.129395 2040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:11.129704 2040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:11.129786 2040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:11.137300 2040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:11.137410 2040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:11.137482 2040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:11.137574 2040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:11.137661 2040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:11.147143 2040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:36485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:36485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:11.148562 2040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:11.150422 2040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:11.162966 2048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:11.163064 2045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:11.163043 2046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:11.164081 2040 server_base.cc:1034] running on GCE node
I20260620 18:49:11.164857 2040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:11.166307 2040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:11.167541 2040 hybrid_clock.cc:648] HybridClock initialized: now 1781981351167510 us; error 55 us; skew 500 ppm
I20260620 18:49:11.167894 2040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:11.170200 2040 webserver.cc:460] Webserver started at http://127.15.102.254:36055/ using document root <none> and password file <none>
I20260620 18:49:11.170915 2040 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:11.171063 2040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:11.171446 2040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:11.174398 2040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "ca2c43c3ed164b319ecc4e1367358eee"
format_stamp: "Formatted at 2026-06-20 18:49:11 on dist-test-slave-zfp8"
server_key: "77c937d2d191c198e467626eecab34bb"
server_key_iv: "73f2e641b228b53b0950cdf3008f2967"
server_key_version: "encryptionkey@0"
I20260620 18:49:11.175223 2040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "ca2c43c3ed164b319ecc4e1367358eee"
format_stamp: "Formatted at 2026-06-20 18:49:11 on dist-test-slave-zfp8"
server_key: "77c937d2d191c198e467626eecab34bb"
server_key_iv: "73f2e641b228b53b0950cdf3008f2967"
server_key_version: "encryptionkey@0"
I20260620 18:49:11.180476 2040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:49:11.184110 2054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:11.185477 2040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:11.185686 2040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "ca2c43c3ed164b319ecc4e1367358eee"
format_stamp: "Formatted at 2026-06-20 18:49:11 on dist-test-slave-zfp8"
server_key: "77c937d2d191c198e467626eecab34bb"
server_key_iv: "73f2e641b228b53b0950cdf3008f2967"
server_key_version: "encryptionkey@0"
I20260620 18:49:11.185915 2040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:11.201256 2040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:11.202102 2040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:11.202402 2040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:11.221004 2040 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:36485
I20260620 18:49:11.221011 2105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:36485 every 8 connection(s)
I20260620 18:49:11.222425 2040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:11.224156 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 2040
I20260620 18:49:11.224359 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:11.224916 15771 external_mini_cluster.cc:1428] Setting key 5de31df8fbbbebb2ce4d4844c6811e91
I20260620 18:49:11.227001 2106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:11.238749 2109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:11.226825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.15.102.254:36485)
0620 18:49:11.227913 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:11.227924 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:11.227964 (+ 40us) server_negotiation.cc:407] Connection header received
0620 18:49:11.229557 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:11.229578 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:11.230044 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:11.230375 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:11.230965 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:11.232370 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:11.234299 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:11.234945 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:11.235093 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:11.236361 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:11.236369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:11.236383 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:11.236949 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:11.237043 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:11.237064 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:11.237154 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:11.237393 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 18:49:11.237768 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":467,"threads_started":1}
I20260620 18:49:11.239952 2106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee: Bootstrap starting.
I20260620 18:49:11.243093 2106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:11.244117 2106 log.cc:824] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:11.246927 2106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee: No bootstrap required, opened a new log
I20260620 18:49:11.254139 2106 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c43c3ed164b319ecc4e1367358eee" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36485 } }
I20260620 18:49:11.254541 2106 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:11.254650 2106 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca2c43c3ed164b319ecc4e1367358eee, State: Initialized, Role: FOLLOWER
I20260620 18:49:11.255205 2106 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c43c3ed164b319ecc4e1367358eee" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36485 } }
I20260620 18:49:11.255523 2106 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:11.255664 2106 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:11.255853 2106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:11.258091 2106 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c43c3ed164b319ecc4e1367358eee" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36485 } }
I20260620 18:49:11.258612 2106 leader_election.cc:302] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca2c43c3ed164b319ecc4e1367358eee; no voters:
I20260620 18:49:11.259270 2106 leader_election.cc:288] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:11.259475 2111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:11.260222 2111 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [term 1 LEADER]: Becoming Leader. State: Replica: ca2c43c3ed164b319ecc4e1367358eee, State: Running, Role: LEADER
I20260620 18:49:11.260769 2111 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c43c3ed164b319ecc4e1367358eee" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36485 } }
I20260620 18:49:11.261404 2106 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:11.263849 2113 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca2c43c3ed164b319ecc4e1367358eee. Latest consensus state: current_term: 1 leader_uuid: "ca2c43c3ed164b319ecc4e1367358eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c43c3ed164b319ecc4e1367358eee" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36485 } } }
I20260620 18:49:11.264322 2113 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:11.265347 2112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca2c43c3ed164b319ecc4e1367358eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca2c43c3ed164b319ecc4e1367358eee" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 36485 } } }
I20260620 18:49:11.265668 2112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:11.267053 2120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:11.272430 2120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:11.281615 2120 catalog_manager.cc:1347] Generated new cluster ID: 5daec9777d67469280e0f75b85f0ca04
I20260620 18:49:11.281739 2120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:11.305476 2120 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:11.306783 2120 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:11.316923 2120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee: Generated new TSK 0
I20260620 18:49:11.317889 2120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:11.336941 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36485
--builtin_ntp_servers=127.15.102.212:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:49:11.669193 2130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:11.669489 2130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:11.669576 2130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:11.676908 2130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:11.677014 2130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:11.677120 2130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:49:11.686342 2130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:11.687628 2130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:11.689467 2130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:11.702582 2136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:11.702517 2138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:11.702983 2135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:11.703513 2130 server_base.cc:1034] running on GCE node
I20260620 18:49:11.704092 2130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:11.705008 2130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:11.706241 2130 hybrid_clock.cc:648] HybridClock initialized: now 1781981351706203 us; error 54 us; skew 500 ppm
I20260620 18:49:11.706557 2130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:11.708761 2130 webserver.cc:460] Webserver started at http://127.15.102.193:46459/ using document root <none> and password file <none>
I20260620 18:49:11.709452 2130 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:11.709579 2130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:11.709893 2130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:11.712776 2130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e"
format_stamp: "Formatted at 2026-06-20 18:49:11 on dist-test-slave-zfp8"
server_key: "879c9ddfa39970f516fd0122db7c25ea"
server_key_iv: "78c62d80be8f3bb6c5290cb0f83ea821"
server_key_version: "encryptionkey@0"
I20260620 18:49:11.713572 2130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e"
format_stamp: "Formatted at 2026-06-20 18:49:11 on dist-test-slave-zfp8"
server_key: "879c9ddfa39970f516fd0122db7c25ea"
server_key_iv: "78c62d80be8f3bb6c5290cb0f83ea821"
server_key_version: "encryptionkey@0"
I20260620 18:49:11.718454 2130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 18:49:11.721843 2145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:11.723073 2130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:49:11.723308 2130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e"
format_stamp: "Formatted at 2026-06-20 18:49:11 on dist-test-slave-zfp8"
server_key: "879c9ddfa39970f516fd0122db7c25ea"
server_key_iv: "78c62d80be8f3bb6c5290cb0f83ea821"
server_key_version: "encryptionkey@0"
I20260620 18:49:11.723519 2130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:11.749111 2130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:11.750018 2130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:11.750411 2130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:11.751575 2130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:11.753202 2130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:11.753335 2130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:11.753481 2130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:11.753593 2130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:11.789916 2130 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:44731
I20260620 18:49:11.789978 2257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:44731 every 8 connection(s)
I20260620 18:49:11.791491 2130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:49:11.801482 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 2130
I20260620 18:49:11.801654 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:49:11.802135 15771 external_mini_cluster.cc:1428] Setting key adb6b7f589b35adf3cd72b08f1560fc0
I20260620 18:49:11.805015 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36485
--builtin_ntp_servers=127.15.102.212:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:49:11.809998 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:11.794341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45705 (local address 127.15.102.254:36485)
0620 18:49:11.794802 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:11.794807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:11.796146 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 18:49:11.798112 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:11.798118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:11.798215 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:11.798349 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:11.799561 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:11.800349 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:11.802170 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:11.802711 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:11.802880 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:11.808850 (+ 5970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:11.808861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:11.808869 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:11.808899 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:11.808961 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:11.808982 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:11.809096 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:11.809371 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 18:49:11.809504 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260620 18:49:11.810859 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:11.794982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36485 (local address 127.15.102.193:45705)
0620 18:49:11.795985 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:11.796025 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:11.797827 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:11.798595 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:11.798604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:11.798706 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:11.799380 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:11.799400 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:11.800511 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:11.800517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:11.802008 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:11.802021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:11.803026 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:11.803033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:11.803368 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:11.808476 (+ 5108us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:11.808479 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:11.808524 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:11.808617 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:11.809102 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:11.809113 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:11.809120 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 18:49:11.809302 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 18:49:11.809635 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":342,"threads_started":1}
I20260620 18:49:11.812693 2258 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36485
I20260620 18:49:11.813084 2258 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:11.814111 2258 heartbeater.cc:511] Master 127.15.102.254:36485 requested a full tablet report, sending...
I20260620 18:49:11.816476 2071 ts_manager.cc:194] Registered new tserver with Master: dd27c3e3d7cf4e04bc7d5b761f8d3a9e (127.15.102.193:44731)
I20260620 18:49:11.818120 2071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.193:45705
W20260620 18:49:12.146263 2262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:12.146584 2262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:12.146750 2262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:12.154398 2262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:12.154544 2262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:12.154718 2262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:49:12.164948 2262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:12.166268 2262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:12.168159 2262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:12.181777 2267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:12.181695 2268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:12.181694 2270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:12.182498 2262 server_base.cc:1034] running on GCE node
I20260620 18:49:12.183314 2262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:12.184449 2262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:12.185705 2262 hybrid_clock.cc:648] HybridClock initialized: now 1781981352185658 us; error 57 us; skew 500 ppm
I20260620 18:49:12.186074 2262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:12.188546 2262 webserver.cc:460] Webserver started at http://127.15.102.194:38861/ using document root <none> and password file <none>
I20260620 18:49:12.189328 2262 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:12.189473 2262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:12.189932 2262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:12.193312 2262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "45b9984e60694b0ab6f859c1dd75c504"
format_stamp: "Formatted at 2026-06-20 18:49:12 on dist-test-slave-zfp8"
server_key: "4251d487428ca5fe9bd8fc3bc7495b7b"
server_key_iv: "e2eeb8f26358bccda908ebff53fb5d72"
server_key_version: "encryptionkey@0"
I20260620 18:49:12.194176 2262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "45b9984e60694b0ab6f859c1dd75c504"
format_stamp: "Formatted at 2026-06-20 18:49:12 on dist-test-slave-zfp8"
server_key: "4251d487428ca5fe9bd8fc3bc7495b7b"
server_key_iv: "e2eeb8f26358bccda908ebff53fb5d72"
server_key_version: "encryptionkey@0"
I20260620 18:49:12.199718 2262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 18:49:12.203321 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:12.205125 2262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:12.205295 2262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "45b9984e60694b0ab6f859c1dd75c504"
format_stamp: "Formatted at 2026-06-20 18:49:12 on dist-test-slave-zfp8"
server_key: "4251d487428ca5fe9bd8fc3bc7495b7b"
server_key_iv: "e2eeb8f26358bccda908ebff53fb5d72"
server_key_version: "encryptionkey@0"
I20260620 18:49:12.205567 2262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:12.245695 2262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:12.246496 2262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:12.246744 2262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:12.247820 2262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:12.249291 2262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:12.249387 2262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:12.249467 2262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:12.249521 2262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:12.282828 2262 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:46355
I20260620 18:49:12.282923 2388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:46355 every 8 connection(s)
I20260620 18:49:12.284157 2262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:49:12.291617 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 2262
I20260620 18:49:12.291801 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:49:12.292243 15771 external_mini_cluster.cc:1428] Setting key 687bfead68a68fd4b1f2d611ed637151
I20260620 18:49:12.295063 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36485
--builtin_ntp_servers=127.15.102.212:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:49:12.297415 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.286819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:41519 (local address 127.15.102.254:36485)
0620 18:49:12.287020 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:12.287024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:12.288380 (+ 1356us) server_negotiation.cc:407] Connection header received
0620 18:49:12.290244 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:12.290249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:12.290317 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:12.290405 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:12.291506 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.292368 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.294533 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.294915 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.295046 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.296578 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:12.296584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:12.296587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:12.296606 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:12.296636 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:12.296648 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:12.296746 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:12.296996 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 18:49:12.297095 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:49:12.298588 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.287331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36485 (local address 127.15.102.194:41519)
0620 18:49:12.288216 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:12.288266 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:12.289962 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:12.290606 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:12.290614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:12.290697 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:12.291334 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.291354 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.292532 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.292538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.294401 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.294412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.295132 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.295138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.295508 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.296241 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:12.296244 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:12.296286 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:12.296363 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:12.296769 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:12.296780 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:12.296788 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 18:49:12.296971 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 18:49:12.297368 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":258,"threads_started":1}
I20260620 18:49:12.299981 2389 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36485
I20260620 18:49:12.300302 2389 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:12.301219 2389 heartbeater.cc:511] Master 127.15.102.254:36485 requested a full tablet report, sending...
I20260620 18:49:12.302747 2071 ts_manager.cc:194] Registered new tserver with Master: 45b9984e60694b0ab6f859c1dd75c504 (127.15.102.194:46355)
I20260620 18:49:12.303613 2071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.194:41519
W20260620 18:49:12.628515 2392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:12.628829 2392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:12.628981 2392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:12.636230 2392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:12.636339 2392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:12.636449 2392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:49:12.645720 2392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:36485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:12.647035 2392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:12.648826 2392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:12.662189 2398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:12.662319 2397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:12.662294 2400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:12.663396 2392 server_base.cc:1034] running on GCE node
I20260620 18:49:12.664078 2392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:12.665223 2392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:12.666462 2392 hybrid_clock.cc:648] HybridClock initialized: now 1781981352666435 us; error 47 us; skew 500 ppm
I20260620 18:49:12.666795 2392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:12.669205 2392 webserver.cc:460] Webserver started at http://127.15.102.195:35413/ using document root <none> and password file <none>
I20260620 18:49:12.669919 2392 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:12.670068 2392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:12.670387 2392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:12.673349 2392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "f9c4f24e25bc4c2d81f27d239a397322"
format_stamp: "Formatted at 2026-06-20 18:49:12 on dist-test-slave-zfp8"
server_key: "cb2b98fe3b18d7dbf700b92471af53c0"
server_key_iv: "b3bd59dda197b31e03e11dc2eabb4cc7"
server_key_version: "encryptionkey@0"
I20260620 18:49:12.674167 2392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "f9c4f24e25bc4c2d81f27d239a397322"
format_stamp: "Formatted at 2026-06-20 18:49:12 on dist-test-slave-zfp8"
server_key: "cb2b98fe3b18d7dbf700b92471af53c0"
server_key_iv: "b3bd59dda197b31e03e11dc2eabb4cc7"
server_key_version: "encryptionkey@0"
I20260620 18:49:12.679388 2392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260620 18:49:12.682806 2406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:12.684201 2392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:12.684381 2392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "f9c4f24e25bc4c2d81f27d239a397322"
format_stamp: "Formatted at 2026-06-20 18:49:12 on dist-test-slave-zfp8"
server_key: "cb2b98fe3b18d7dbf700b92471af53c0"
server_key_iv: "b3bd59dda197b31e03e11dc2eabb4cc7"
server_key_version: "encryptionkey@0"
I20260620 18:49:12.684602 2392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:12.702258 2392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:12.703325 2392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:12.703730 2392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:12.704797 2392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:12.706363 2392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:12.706493 2392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:12.706636 2392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:12.706750 2392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:12.742291 2392 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:45627
I20260620 18:49:12.742336 2518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:45627 every 8 connection(s)
I20260620 18:49:12.743770 2392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:49:12.751444 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 2392
I20260620 18:49:12.751663 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:49:12.752164 15771 external_mini_cluster.cc:1428] Setting key e101b2d41132fdf1dd2a930e5b8579ea
I20260620 18:49:12.756724 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.746396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:42645 (local address 127.15.102.254:36485)
0620 18:49:12.746601 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:12.746605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:12.748117 (+ 1512us) server_negotiation.cc:407] Connection header received
0620 18:49:12.750027 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:12.750033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:12.750127 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:12.750236 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:12.751606 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.752490 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.754021 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.754462 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.754599 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.755921 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:12.755933 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:12.755937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:12.755954 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:12.755985 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:12.755997 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:12.756093 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:12.756315 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 18:49:12.756410 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:49:12.757807 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.747013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:36485 (local address 127.15.102.195:42645)
0620 18:49:12.747962 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:12.747997 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:12.749745 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:12.750506 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:12.750522 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:12.750614 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:12.751418 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.751438 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.752640 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.752646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.753872 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.753884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.754690 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.754697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.754935 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.755607 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:12.755610 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:12.755647 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:12.755729 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:12.756091 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:12.756103 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:12.756113 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 18:49:12.756290 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 18:49:12.756689 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":336,"threads_started":1}
I20260620 18:49:12.759037 2519 heartbeater.cc:347] Connected to a master server at 127.15.102.254:36485
I20260620 18:49:12.759377 2519 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:12.760131 2519 heartbeater.cc:511] Master 127.15.102.254:36485 requested a full tablet report, sending...
I20260620 18:49:12.761478 2071 ts_manager.cc:194] Registered new tserver with Master: f9c4f24e25bc4c2d81f27d239a397322 (127.15.102.195:45627)
I20260620 18:49:12.762389 2071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.195:42645
I20260620 18:49:12.767067 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:49:12.776782 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.771032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.15.102.254:36485)
0620 18:49:12.771309 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:12.771317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:12.771642 (+ 325us) server_negotiation.cc:407] Connection header received
0620 18:49:12.771766 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:12.771769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:12.771876 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:12.771957 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:12.772403 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.773106 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.774394 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.774799 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.774894 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.776022 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:12.776028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:12.776033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:12.776055 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:12.776094 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:12.776111 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:12.776273 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:12.776413 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:49:12.776487 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:49:12.780552 2071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:49:12.787796 2071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:49:12.811686 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.800453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.0.0.1:60256)
0620 18:49:12.801773 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:12.801807 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:12.801949 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:12.802959 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:12.802966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:12.802997 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:12.803442 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.803461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.805840 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.805848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.808481 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.808497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.810359 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.810366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.810613 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.810678 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 18:49:12.810832 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 18:49:12.811219 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":121,"thread_start_us":1001,"threads_started":1}
I20260620 18:49:12.811887 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.800596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.15.102.193:44731)
0620 18:49:12.801324 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:12.801336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:12.801920 (+ 584us) server_negotiation.cc:407] Connection header received
0620 18:49:12.802148 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:12.802156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:12.802443 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:12.802653 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:12.803662 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.805649 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.808690 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.810164 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.810376 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.810650 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:12.810894 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 18:49:12.811087 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":353,"threads_started":1}
I20260620 18:49:12.815595 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.802228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:45627 (local address 127.0.0.1:52340)
0620 18:49:12.802775 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:12.802792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:12.802935 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:12.807974 (+ 5039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:12.807980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:12.808018 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:12.808755 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.808767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.810085 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.810091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.813326 (+ 3235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.813341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.814709 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.814717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.814896 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.814931 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 18:49:12.815034 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 18:49:12.815165 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":131,"spinlock_wait_cycles":384,"thread_start_us":273,"threads_started":1}
I20260620 18:49:12.815634 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.802581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.15.102.195:45627)
0620 18:49:12.807130 (+ 4549us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:12.807138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:12.807166 (+ 28us) server_negotiation.cc:407] Connection header received
0620 18:49:12.807290 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:12.807296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:12.807505 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:12.807699 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:12.808966 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.809923 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.813551 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.814508 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.814728 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.814955 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:12.815071 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 18:49:12.815264 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4363,"thread_start_us":4199,"threads_started":1}
I20260620 18:49:12.816100 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.802799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:46355 (local address 127.0.0.1:43342)
0620 18:49:12.806757 (+ 3958us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:12.806782 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:12.806914 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:12.807974 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:12.807980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:12.808024 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:12.808596 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.808615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.810495 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.810501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.813102 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:12.813116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.815272 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.815280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:12.815457 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.815493 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 18:49:12.815591 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:49:12.815720 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3812,"mutex_wait_us":2,"spinlock_wait_cycles":6784,"thread_start_us":3658,"threads_started":1}
I20260620 18:49:12.816823 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:12.803277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.15.102.194:46355)
0620 18:49:12.806228 (+ 2951us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:12.806241 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:12.807062 (+ 821us) server_negotiation.cc:407] Connection header received
0620 18:49:12.807174 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:12.807184 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:12.807502 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:12.807699 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:12.808794 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.810290 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.813309 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:12.814746 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:12.814910 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:12.815107 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:12.816312 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0620 18:49:12.816480 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2789,"thread_start_us":2679,"threads_started":1}
I20260620 18:49:12.816937 2193 tablet_service.cc:1467] Processing CreateTablet for tablet 3421f4a488b64952bcdbd70a46f9aa10 (DEFAULT_TABLE table=test-table [id=56b63ae280034ddbb8d34819b71ee21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:12.818943 2193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3421f4a488b64952bcdbd70a46f9aa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:12.819921 2454 tablet_service.cc:1467] Processing CreateTablet for tablet 3421f4a488b64952bcdbd70a46f9aa10 (DEFAULT_TABLE table=test-table [id=56b63ae280034ddbb8d34819b71ee21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:12.821327 2258 heartbeater.cc:503] Master 127.15.102.254:36485 was elected leader, sending a full tablet report...
I20260620 18:49:12.821390 2454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3421f4a488b64952bcdbd70a46f9aa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:12.822294 2324 tablet_service.cc:1467] Processing CreateTablet for tablet 3421f4a488b64952bcdbd70a46f9aa10 (DEFAULT_TABLE table=test-table [id=56b63ae280034ddbb8d34819b71ee21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:12.823603 2324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3421f4a488b64952bcdbd70a46f9aa10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:12.834728 2537 tablet_bootstrap.cc:492] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Bootstrap starting.
I20260620 18:49:12.837988 2537 tablet_bootstrap.cc:654] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:12.838930 2538 tablet_bootstrap.cc:492] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e: Bootstrap starting.
I20260620 18:49:12.839401 2537 log.cc:824] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:12.841966 2539 tablet_bootstrap.cc:492] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Bootstrap starting.
I20260620 18:49:12.843222 2538 tablet_bootstrap.cc:654] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:12.845000 2539 tablet_bootstrap.cc:654] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:12.845436 2538 log.cc:824] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:12.845965 2539 log.cc:824] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:12.849202 2539 tablet_bootstrap.cc:492] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: No bootstrap required, opened a new log
I20260620 18:49:12.849237 2537 tablet_bootstrap.cc:492] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: No bootstrap required, opened a new log
I20260620 18:49:12.849550 2539 ts_tablet_manager.cc:1397] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 18:49:12.849536 2537 ts_tablet_manager.cc:1397] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260620 18:49:12.849753 2538 tablet_bootstrap.cc:492] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e: No bootstrap required, opened a new log
I20260620 18:49:12.850011 2538 ts_tablet_manager.cc:1397] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260620 18:49:12.857550 2537 raft_consensus.cc:348] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:12.857939 2537 raft_consensus.cc:374] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:12.858059 2537 raft_consensus.cc:729] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b9984e60694b0ab6f859c1dd75c504, State: Initialized, Role: FOLLOWER
I20260620 18:49:12.858681 2537 consensus_queue.cc:260] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:12.859797 2389 heartbeater.cc:503] Master 127.15.102.254:36485 was elected leader, sending a full tablet report...
I20260620 18:49:12.860468 2539 raft_consensus.cc:348] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:12.860482 2538 raft_consensus.cc:348] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:12.860908 2538 raft_consensus.cc:374] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:12.861001 2539 raft_consensus.cc:374] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:12.861027 2538 raft_consensus.cc:729] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd27c3e3d7cf4e04bc7d5b761f8d3a9e, State: Initialized, Role: FOLLOWER
I20260620 18:49:12.861119 2539 raft_consensus.cc:729] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c4f24e25bc4c2d81f27d239a397322, State: Initialized, Role: FOLLOWER
I20260620 18:49:12.861794 2537 ts_tablet_manager.cc:1428] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260620 18:49:12.861686 2538 consensus_queue.cc:260] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:12.861691 2539 consensus_queue.cc:260] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:12.862936 2519 heartbeater.cc:503] Master 127.15.102.254:36485 was elected leader, sending a full tablet report...
I20260620 18:49:12.863772 2539 ts_tablet_manager.cc:1428] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260620 18:49:12.866377 2538 ts_tablet_manager.cc:1428] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
W20260620 18:49:12.995656 2520 tablet.cc:2369] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:49:13.036366 2390 tablet.cc:2369] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:49:13.044353 2259 tablet.cc:2369] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:13.078040 2543 raft_consensus.cc:482] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:49:13.078373 2543 raft_consensus.cc:504] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:13.080137 2543 leader_election.cc:288] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd27c3e3d7cf4e04bc7d5b761f8d3a9e (127.15.102.193:44731), f9c4f24e25bc4c2d81f27d239a397322 (127.15.102.195:45627)
I20260620 18:49:13.088987 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.081686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:33163 (local address 127.15.102.195:45627)
0620 18:49:13.081897 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:13.081902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:13.082162 (+ 260us) server_negotiation.cc:407] Connection header received
0620 18:49:13.082317 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:13.082320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:13.082376 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:13.082450 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:13.082948 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.083970 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.086631 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.087955 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.088105 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:13.088237 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:13.088614 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 18:49:13.088741 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 18:49:13.089049 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.080686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:47299 (local address 127.15.102.193:44731)
0620 18:49:13.080884 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:13.080889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:13.081403 (+ 514us) server_negotiation.cc:407] Connection header received
0620 18:49:13.081617 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:13.081623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:13.081756 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:13.081874 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:13.082604 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.083703 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.086637 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.087931 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.088105 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:13.088180 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:13.088614 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 18:49:13.088752 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:49:13.089097 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.081408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:45627 (local address 127.15.102.194:33163)
0620 18:49:13.082049 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.082067 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:13.082199 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:13.082500 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:13.082506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:13.082534 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:13.082803 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:13.082811 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.084096 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.084101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:13.086379 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:13.086395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.088197 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.088203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:13.088427 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:13.088475 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:49:13.088574 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 18:49:13.088736 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":60,"thread_start_us":318,"threads_started":1}
I20260620 18:49:13.089236 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.080552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.15.102.194:47299)
0620 18:49:13.081282 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.081303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:13.081452 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:13.081953 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:13.081959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:13.081985 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:13.082411 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:13.082451 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.083844 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.083847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:13.086386 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:13.086395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.088190 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.088197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:13.088441 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:13.088482 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 18:49:13.088574 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 18:49:13.088704 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":213,"thread_start_us":409,"threads_started":1}
I20260620 18:49:13.090451 2213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3421f4a488b64952bcdbd70a46f9aa10" candidate_uuid: "45b9984e60694b0ab6f859c1dd75c504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" is_pre_election: true
I20260620 18:49:13.091038 2213 raft_consensus.cc:2454] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b9984e60694b0ab6f859c1dd75c504 in term 0.
I20260620 18:49:13.091187 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3421f4a488b64952bcdbd70a46f9aa10" candidate_uuid: "45b9984e60694b0ab6f859c1dd75c504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c4f24e25bc4c2d81f27d239a397322" is_pre_election: true
I20260620 18:49:13.091758 2474 raft_consensus.cc:2454] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45b9984e60694b0ab6f859c1dd75c504 in term 0.
I20260620 18:49:13.091760 2277 leader_election.cc:302] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b9984e60694b0ab6f859c1dd75c504, dd27c3e3d7cf4e04bc7d5b761f8d3a9e; no voters:
I20260620 18:49:13.092168 2543 raft_consensus.cc:2781] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:49:13.092298 2543 raft_consensus.cc:482] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:49:13.092372 2543 raft_consensus.cc:3037] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:13.094456 2543 raft_consensus.cc:504] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:13.095165 2543 leader_election.cc:288] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [CANDIDATE]: Term 1 election: Requested vote from peers dd27c3e3d7cf4e04bc7d5b761f8d3a9e (127.15.102.193:44731), f9c4f24e25bc4c2d81f27d239a397322 (127.15.102.195:45627)
I20260620 18:49:13.095575 2213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3421f4a488b64952bcdbd70a46f9aa10" candidate_uuid: "45b9984e60694b0ab6f859c1dd75c504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e"
I20260620 18:49:13.095645 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3421f4a488b64952bcdbd70a46f9aa10" candidate_uuid: "45b9984e60694b0ab6f859c1dd75c504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9c4f24e25bc4c2d81f27d239a397322"
I20260620 18:49:13.095794 2213 raft_consensus.cc:3037] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:13.095819 2474 raft_consensus.cc:3037] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:13.098977 2474 raft_consensus.cc:2454] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b9984e60694b0ab6f859c1dd75c504 in term 1.
I20260620 18:49:13.098977 2213 raft_consensus.cc:2454] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45b9984e60694b0ab6f859c1dd75c504 in term 1.
I20260620 18:49:13.099360 2277 leader_election.cc:302] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45b9984e60694b0ab6f859c1dd75c504, f9c4f24e25bc4c2d81f27d239a397322; no voters:
I20260620 18:49:13.099653 2543 raft_consensus.cc:2781] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:13.100106 2543 raft_consensus.cc:686] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 1 LEADER]: Becoming Leader. State: Replica: 45b9984e60694b0ab6f859c1dd75c504, State: Running, Role: LEADER
I20260620 18:49:13.100584 2543 consensus_queue.cc:237] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } }
I20260620 18:49:13.105329 2070 catalog_manager.cc:5495] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45b9984e60694b0ab6f859c1dd75c504 (127.15.102.194). New cstate: current_term: 1 leader_uuid: "45b9984e60694b0ab6f859c1dd75c504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45b9984e60694b0ab6f859c1dd75c504" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 46355 } health_report { overall_health: HEALTHY } } }
I20260620 18:49:13.179049 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.174000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.15.102.194:46355)
0620 18:49:13.174167 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:13.174171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:13.174191 (+ 20us) server_negotiation.cc:407] Connection header received
0620 18:49:13.174388 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:13.174391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:13.174489 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:13.174611 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:13.174950 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.175711 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.177031 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.177453 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.177547 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:13.178035 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:13.178524 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:13.178653 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:13.178803 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 18:49:13.178840 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 18:49:13.187175 2474 raft_consensus.cc:1261] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 1 FOLLOWER]: Refusing update from remote peer 45b9984e60694b0ab6f859c1dd75c504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:13.187175 2213 raft_consensus.cc:1261] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 1 FOLLOWER]: Refusing update from remote peer 45b9984e60694b0ab6f859c1dd75c504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:13.188218 2548 consensus_queue.cc:1035] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9c4f24e25bc4c2d81f27d239a397322" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 45627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:13.188563 2543 consensus_queue.cc:1035] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd27c3e3d7cf4e04bc7d5b761f8d3a9e" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:13.201085 2553 mvcc.cc:205] Tried to move back new op lower bound from 7298995622645108736 to 7298995622303920128. Current Snapshot: MvccSnapshot[applied={T|T < 7298995622645108736}]
I20260620 18:49:13.220594 2071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:13.220891 2071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:13.224987 2071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca2c43c3ed164b319ecc4e1367358eee: Sending DeleteTablet for 3 replicas of tablet 3421f4a488b64952bcdbd70a46f9aa10
I20260620 18:49:13.226202 2454 tablet_service.cc:1514] Processing DeleteTablet for tablet 3421f4a488b64952bcdbd70a46f9aa10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:13 UTC) from {username='slave'} at 127.0.0.1:52340
I20260620 18:49:13.226292 2193 tablet_service.cc:1514] Processing DeleteTablet for tablet 3421f4a488b64952bcdbd70a46f9aa10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:13 UTC) from {username='slave'} at 127.0.0.1:60256
I20260620 18:49:13.226430 2324 tablet_service.cc:1514] Processing DeleteTablet for tablet 3421f4a488b64952bcdbd70a46f9aa10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:13 UTC) from {username='slave'} at 127.0.0.1:43342
I20260620 18:49:13.228046 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 2130
I20260620 18:49:13.250504 2561 tablet_replica.cc:330] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: stopping tablet replica
I20260620 18:49:13.251116 2561 raft_consensus.cc:2229] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:13.251753 2561 raft_consensus.cc:2258] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:13.254160 2561 ts_tablet_manager.cc:1905] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:13.258713 2563 tablet_replica.cc:330] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: stopping tablet replica
I20260620 18:49:13.259152 2563 raft_consensus.cc:2229] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:49:13.259802 2563 raft_consensus.cc:2258] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:13.263995 2563 ts_tablet_manager.cc:1905] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:13.265936 2562 tablet_replica.cc:330] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e: stopping tablet replica
I20260620 18:49:13.266332 2562 raft_consensus.cc:2229] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:13.266806 2562 raft_consensus.cc:2258] T 3421f4a488b64952bcdbd70a46f9aa10 P dd27c3e3d7cf4e04bc7d5b761f8d3a9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:13.267696 2561 ts_tablet_manager.cc:1918] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:13.267859 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.229366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.15.102.193:44731)
0620 18:49:13.229526 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:13.229532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:13.253567 (+ 24035us) server_negotiation.cc:407] Connection header received
0620 18:49:13.254063 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:13.254069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:13.254201 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:13.254291 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:13.254984 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.256184 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.260170 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.260829 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.260924 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:13.264442 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:13.264454 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:13.264472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:13.265238 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:13.265307 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:13.265331 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:13.265437 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:13.267392 (+ 1955us) server_negotiation.cc:299] Negotiation successful
0620 18:49:13.267519 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:49:13.267874 2561 log.cc:1196] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/3421f4a488b64952bcdbd70a46f9aa10
I20260620 18:49:13.268386 2561 ts_tablet_manager.cc:1939] T 3421f4a488b64952bcdbd70a46f9aa10 P f9c4f24e25bc4c2d81f27d239a397322: Deleting consensus metadata
I20260620 18:49:13.269140 2253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:13.277104 2563 ts_tablet_manager.cc:1918] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:13.277279 2563 log.cc:1196] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/3421f4a488b64952bcdbd70a46f9aa10
I20260620 18:49:13.277770 2563 ts_tablet_manager.cc:1939] T 3421f4a488b64952bcdbd70a46f9aa10 P 45b9984e60694b0ab6f859c1dd75c504: Deleting consensus metadata
I20260620 18:49:13.279596 2055 catalog_manager.cc:4841] TS 45b9984e60694b0ab6f859c1dd75c504 (127.15.102.194:46355): tablet 3421f4a488b64952bcdbd70a46f9aa10 (table test-table [id=56b63ae280034ddbb8d34819b71ee21d]) successfully deleted
I20260620 18:49:13.280766 2055 catalog_manager.cc:4841] TS f9c4f24e25bc4c2d81f27d239a397322 (127.15.102.195:45627): tablet 3421f4a488b64952bcdbd70a46f9aa10 (table test-table [id=56b63ae280034ddbb8d34819b71ee21d]) successfully deleted
I20260620 18:49:13.389158 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 2130
I20260620 18:49:13.406946 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 2262
W20260620 18:49:13.406942 2055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:44731 (error 108)
I20260620 18:49:13.411686 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.408493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.0.0.1:60262)
0620 18:49:13.411092 (+ 2599us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.411462 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2465,"thread_start_us":2360,"threads_started":1}
W20260620 18:49:13.412192 2055 catalog_manager.cc:4584] TS dd27c3e3d7cf4e04bc7d5b761f8d3a9e (127.15.102.193:44731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3421f4a488b64952bcdbd70a46f9aa10: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44731: connect: Connection refused (error 111)
I20260620 18:49:13.416256 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.408244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.15.102.194:46355)
0620 18:49:13.408468 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:13.408474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:13.408493 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:49:13.408565 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:13.408573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:13.408701 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:13.408861 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:13.409699 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.411160 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.412951 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.413394 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.413552 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:13.414717 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:13.414740 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:13.414754 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:13.415465 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:13.415515 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:13.415535 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:13.415675 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:13.415821 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 18:49:13.415924 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 18:49:13.416997 2384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:13.465919 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.465434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.0.0.1:60276)
0620 18:49:13.465654 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.465752 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:49:13.467119 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.466723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.0.0.1:60284)
0620 18:49:13.466889 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.466984 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 18:49:13.527843 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.527370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.0.0.1:60300)
0620 18:49:13.527589 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.527678 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:49:13.528889 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.528518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.0.0.1:60302)
0620 18:49:13.528673 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.528753 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 18:49:13.533821 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 2262
I20260620 18:49:13.553232 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 2392
I20260620 18:49:13.560583 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.554008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.15.102.195:45627)
0620 18:49:13.554192 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:13.554198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:13.554220 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:49:13.554355 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:13.554360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:13.554450 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:13.554558 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:13.554981 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.556033 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.557430 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:13.557866 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:13.558000 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:13.559125 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:13.559134 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:13.559149 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:13.559796 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:13.559844 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:13.559869 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:13.559992 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:13.560127 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:49:13.560236 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:49:13.561303 2514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:13.609382 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.608889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.0.0.1:60308)
0620 18:49:13.609133 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.609227 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 18:49:13.610481 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:13.610061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:44731 (local address 127.0.0.1:60324)
0620 18:49:13.610266 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:13.610340 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:44731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 18:49:13.650055 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 2392
I20260620 18:49:13.667215 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 2040
I20260620 18:49:13.668385 2101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:13.778522 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 2040
2026-06-20T18:49:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3069 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:13 dist-test-slave-zfp8 krb5kdc[2573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:13 dist-test-slave-zfp8 krb5kdc[2573](info): set up 2 sockets
Jun 20 18:49:13 dist-test-slave-zfp8 krb5kdc[2573](info): commencing operation
krb5kdc: starting...
W20260620 18:49:14.899641 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:15 dist-test-slave-zfp8 krb5kdc[2573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:15.153053 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:42987
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:42987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:15.487341 2586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:15.487713 2586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:15.487890 2586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:15.495126 2586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:15.495325 2586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:15.495446 2586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:15.495559 2586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:15.495649 2586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:15.505661 2586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:42987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:42987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:15.507064 2586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:15.508435 2586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:15.520349 2594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:15.520401 2591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:15.520277 2592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:15.521473 2586 server_base.cc:1034] running on GCE node
I20260620 18:49:15.522277 2586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:15.523831 2586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:15.525094 2586 hybrid_clock.cc:648] HybridClock initialized: now 1781981355525030 us; error 47 us; skew 500 ppm
I20260620 18:49:15.525456 2586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:15 dist-test-slave-zfp8 krb5kdc[2573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:15.532872 2586 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:15.534361 2586 webserver.cc:460] Webserver started at http://127.15.102.254:43607/ using document root <none> and password file <none>
I20260620 18:49:15.535012 2586 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:15.535109 2586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:15.535439 2586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:15.538425 2586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "5f122b0327d24e75a90adaea03eda733"
format_stamp: "Formatted at 2026-06-20 18:49:15 on dist-test-slave-zfp8"
server_key: "b7f4ef575cf1fd8c9508cbbb0a599029"
server_key_iv: "3e7686cea151fc97f93076956d2ff249"
server_key_version: "encryptionkey@0"
I20260620 18:49:15.539137 2586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "5f122b0327d24e75a90adaea03eda733"
format_stamp: "Formatted at 2026-06-20 18:49:15 on dist-test-slave-zfp8"
server_key: "b7f4ef575cf1fd8c9508cbbb0a599029"
server_key_iv: "3e7686cea151fc97f93076956d2ff249"
server_key_version: "encryptionkey@0"
I20260620 18:49:15.544435 2586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 18:49:15.547926 2601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:15.549188 2586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:15.549364 2586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "5f122b0327d24e75a90adaea03eda733"
format_stamp: "Formatted at 2026-06-20 18:49:15 on dist-test-slave-zfp8"
server_key: "b7f4ef575cf1fd8c9508cbbb0a599029"
server_key_iv: "3e7686cea151fc97f93076956d2ff249"
server_key_version: "encryptionkey@0"
I20260620 18:49:15.549546 2586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:15.573150 2586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:15.577970 2586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:15.578289 2586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:15.598124 2586 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:42987
I20260620 18:49:15.598137 2652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:42987 every 8 connection(s)
I20260620 18:49:15.599593 2586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:15.603404 2653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:15.608125 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 2586
I20260620 18:49:15.608346 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:15.608793 15771 external_mini_cluster.cc:1428] Setting key 9ddec57d76dbd7a6bf22e1912073ba03
I20260620 18:49:15.614382 2653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733: Bootstrap starting.
I20260620 18:49:15.618194 2653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:15.619493 2653 log.cc:824] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:15.623104 2653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733: No bootstrap required, opened a new log
Jun 20 18:49:15 dist-test-slave-zfp8 krb5kdc[2573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:15.630939 2653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f122b0327d24e75a90adaea03eda733" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42987 } }
I20260620 18:49:15.631376 2653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:15.631518 2653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f122b0327d24e75a90adaea03eda733, State: Initialized, Role: FOLLOWER
I20260620 18:49:15.632184 2653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f122b0327d24e75a90adaea03eda733" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42987 } }
I20260620 18:49:15.632484 2653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:15.632663 2653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:15.632861 2653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:15.633240 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:15.610751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.15.102.254:42987)
0620 18:49:15.611628 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:15.611639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:15.611683 (+ 44us) server_negotiation.cc:407] Connection header received
0620 18:49:15.613423 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:15.613448 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:15.613845 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:15.614177 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:15.616068 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:15.617437 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:15.619435 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:15.620505 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:15.620719 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:15.625472 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:15.625480 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:15.625498 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:15.625535 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:15.628832 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:15.629573 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:15.629580 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:15.629586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:15.629762 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:15.630178 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:15.630181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:15.630183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:15.630700 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:15.631009 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:15.631520 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 18:49:15.632011 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":428,"threads_started":1}
I20260620 18:49:15.635349 2653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f122b0327d24e75a90adaea03eda733" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42987 } }
I20260620 18:49:15.635880 2653 leader_election.cc:302] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f122b0327d24e75a90adaea03eda733; no voters:
I20260620 18:49:15.636518 2653 leader_election.cc:288] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:15.636674 2658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:15.637362 2658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [term 1 LEADER]: Becoming Leader. State: Replica: 5f122b0327d24e75a90adaea03eda733, State: Running, Role: LEADER
I20260620 18:49:15.637866 2658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f122b0327d24e75a90adaea03eda733" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42987 } }
I20260620 18:49:15.638633 2653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:15.640564 2659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f122b0327d24e75a90adaea03eda733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f122b0327d24e75a90adaea03eda733" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42987 } } }
I20260620 18:49:15.640893 2659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:15.641386 2660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f122b0327d24e75a90adaea03eda733. Latest consensus state: current_term: 1 leader_uuid: "5f122b0327d24e75a90adaea03eda733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f122b0327d24e75a90adaea03eda733" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42987 } } }
I20260620 18:49:15.641695 2660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:15.641911 2664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:15.648939 2664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:15.660320 2664 catalog_manager.cc:1347] Generated new cluster ID: 596a8a816b774d61b6bb86d5ff55b6c1
I20260620 18:49:15.660450 2664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:15.669744 2664 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:15.671654 2664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:15.680197 2664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733: Generated new TSK 0
I20260620 18:49:15.681234 2664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.102.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
Entry for principal kudu/127.15.102.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab.
I20260620 18:49:15.744860 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:42987
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:16.085258 2680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:16.085547 2680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:16.085640 2680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:16.092780 2680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:16.092918 2680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:16.093030 2680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:49:16.102555 2680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.193.keytab
--principal=kudu/127.15.102.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:42987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:16.103845 2680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:16.105098 2680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:16.118420 2688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:16.118474 2685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:16.118443 2686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:16.119724 2680 server_base.cc:1034] running on GCE node
I20260620 18:49:16.120353 2680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:16.121433 2680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:16.122661 2680 hybrid_clock.cc:648] HybridClock initialized: now 1781981356122622 us; error 32 us; skew 500 ppm
I20260620 18:49:16.122942 2680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:16 dist-test-slave-zfp8 krb5kdc[2573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981356, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:16.126236 2680 init.cc:377] Logged in from keytab as kudu/127.15.102.193@KRBTEST.COM (short username kudu)
I20260620 18:49:16.127715 2680 webserver.cc:460] Webserver started at http://127.15.102.193:36981/ using document root <none> and password file <none>
I20260620 18:49:16.128353 2680 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:16.128455 2680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:16.128720 2680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:16.131734 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "f9268de3fe5e4be8b46e45e7766308d5"
format_stamp: "Formatted at 2026-06-20 18:49:16 on dist-test-slave-zfp8"
server_key: "c38d7d87bcee331eda791b0a6e7dc0c8"
server_key_iv: "004836c0e59f4fe74e569b009b220e7f"
server_key_version: "encryptionkey@0"
I20260620 18:49:16.132629 2680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "f9268de3fe5e4be8b46e45e7766308d5"
format_stamp: "Formatted at 2026-06-20 18:49:16 on dist-test-slave-zfp8"
server_key: "c38d7d87bcee331eda791b0a6e7dc0c8"
server_key_iv: "004836c0e59f4fe74e569b009b220e7f"
server_key_version: "encryptionkey@0"
I20260620 18:49:16.138005 2680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:49:16.141819 2695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:16.143326 2680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:49:16.143501 2680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "f9268de3fe5e4be8b46e45e7766308d5"
format_stamp: "Formatted at 2026-06-20 18:49:16 on dist-test-slave-zfp8"
server_key: "c38d7d87bcee331eda791b0a6e7dc0c8"
server_key_iv: "004836c0e59f4fe74e569b009b220e7f"
server_key_version: "encryptionkey@0"
I20260620 18:49:16.143733 2680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:16.156447 2680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:16.161026 2680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:16.161331 2680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:16.162338 2680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:16.163932 2680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:16.164007 2680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:16.164189 2680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:16.164254 2680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:16.199038 2680 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:43251
I20260620 18:49:16.199074 2808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:43251 every 8 connection(s)
I20260620 18:49:16.200368 2680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:49:16.201337 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 2680
I20260620 18:49:16.201524 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:49:16.202068 15771 external_mini_cluster.cc:1428] Setting key e9a757ad96c41934f05331204457eae2
Jun 20 18:49:16 dist-test-slave-zfp8 krb5kdc[2573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981356, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.193@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:16.225452 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:16.203419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:55801 (local address 127.15.102.254:42987)
0620 18:49:16.203895 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:16.203900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:16.205233 (+ 1333us) server_negotiation.cc:407] Connection header received
0620 18:49:16.208141 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:16.208146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:16.208213 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:16.208314 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:16.210443 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:16.211165 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:16.213744 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:16.214257 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:16.214429 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:16.218127 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:16.218136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:16.218143 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:16.218189 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:16.221884 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:16.222659 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:16.222668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:16.222674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:16.222825 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:16.223374 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:16.223382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:16.223389 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:16.223766 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:16.223921 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:16.224680 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 18:49:16.224939 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260620 18:49:16.226121 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:16.204003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:42987 (local address 127.15.102.193:55801)
0620 18:49:16.205030 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:16.205083 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:16.207841 (+ 2758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:16.208588 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:16.208602 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:16.209408 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:16.210247 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:16.210274 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:16.211372 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:16.211380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:16.213571 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:16.213588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:16.214513 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:16.214517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:16.214729 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:16.215506 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:16.215528 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:16.217898 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:16.222055 (+ 4157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:16.222062 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:16.222075 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:16.222517 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:16.222973 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:16.222976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:16.222979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:16.223177 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:16.223903 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:16.223909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:16.224269 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 18:49:16.224500 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 18:49:16.224910 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":370,"threads_started":1}
I20260620 18:49:16.227963 2809 heartbeater.cc:347] Connected to a master server at 127.15.102.254:42987
I20260620 18:49:16.228307 2809 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:16.229063 2809 heartbeater.cc:511] Master 127.15.102.254:42987 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.102.194@KRBTEST.COM; defaulting to no policy
I20260620 18:49:16.231292 2618 ts_manager.cc:194] Registered new tserver with Master: f9268de3fe5e4be8b46e45e7766308d5 (127.15.102.193:43251)
I20260620 18:49:16.232941 2618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.193@KRBTEST.COM'} at 127.15.102.193:55801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
Entry for principal kudu/127.15.102.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab.
I20260620 18:49:16.263777 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:42987
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:16.594429 2815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:16.594714 2815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:16.594899 2815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:16.602355 2815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:16.602471 2815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:16.602663 2815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:49:16.612888 2815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.194.keytab
--principal=kudu/127.15.102.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:42987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:16.614230 2815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:16.615669 2815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:16.629379 2821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:16.629304 2823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:16.629343 2820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:16.630234 2815 server_base.cc:1034] running on GCE node
I20260620 18:49:16.630879 2815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:16.631945 2815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:16.633185 2815 hybrid_clock.cc:648] HybridClock initialized: now 1781981356633128 us; error 56 us; skew 500 ppm
I20260620 18:49:16.633514 2815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:16 dist-test-slave-zfp8 krb5kdc[2573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981356, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:16.636695 2815 init.cc:377] Logged in from keytab as kudu/127.15.102.194@KRBTEST.COM (short username kudu)
I20260620 18:49:16.638345 2815 webserver.cc:460] Webserver started at http://127.15.102.194:36173/ using document root <none> and password file <none>
I20260620 18:49:16.639068 2815 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:16.639171 2815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:16.639605 2815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:16.642561 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0"
format_stamp: "Formatted at 2026-06-20 18:49:16 on dist-test-slave-zfp8"
server_key: "93e66611e1fcce6fccb5e4d5b2d31649"
server_key_iv: "d4e28df29035f04781dd256f4851a239"
server_key_version: "encryptionkey@0"
I20260620 18:49:16.643407 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0"
format_stamp: "Formatted at 2026-06-20 18:49:16 on dist-test-slave-zfp8"
server_key: "93e66611e1fcce6fccb5e4d5b2d31649"
server_key_iv: "d4e28df29035f04781dd256f4851a239"
server_key_version: "encryptionkey@0"
I20260620 18:49:16.648782 2815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:49:16.652518 2830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:16.654050 2815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:16.654227 2815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0"
format_stamp: "Formatted at 2026-06-20 18:49:16 on dist-test-slave-zfp8"
server_key: "93e66611e1fcce6fccb5e4d5b2d31649"
server_key_iv: "d4e28df29035f04781dd256f4851a239"
server_key_version: "encryptionkey@0"
I20260620 18:49:16.654455 2815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:16.672312 2815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:16.676545 2815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:16.676862 2815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:16.677950 2815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:16.679399 2815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:16.679471 2815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:16.679653 2815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:16.679705 2815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:16.714174 2815 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:33657
I20260620 18:49:16.714274 2942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:33657 every 8 connection(s)
I20260620 18:49:16.715613 2815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:49:16.718674 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 2815
I20260620 18:49:16.718871 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:49:16.719409 15771 external_mini_cluster.cc:1428] Setting key b9cc4c3bcbd6e445e69fceff98f93c63
Jun 20 18:49:16 dist-test-slave-zfp8 krb5kdc[2573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981356, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.194@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:16.740315 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:16.718751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:46963 (local address 127.15.102.254:42987)
0620 18:49:16.718981 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:16.718988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:16.720506 (+ 1518us) server_negotiation.cc:407] Connection header received
0620 18:49:16.722468 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:16.722477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:16.722569 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:16.722680 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:16.724399 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:16.725525 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:16.727512 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:16.728252 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:16.728402 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:16.732869 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:16.732878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:16.732885 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:16.732933 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:16.736515 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:16.737321 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:16.737329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:16.737335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:16.737502 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:16.738030 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:16.738037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:16.738043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:16.738460 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:16.738618 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:16.739593 (+ 975us) server_negotiation.cc:299] Negotiation successful
0620 18:49:16.739855 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 18:49:16.741106 2945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:16.719389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:42987 (local address 127.15.102.194:46963)
0620 18:49:16.720326 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:16.720382 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:16.722167 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:16.722932 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:16.722942 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:16.723585 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:16.724204 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:16.724223 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:16.725722 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:16.725727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:16.727353 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:16.727368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:16.728544 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:16.728550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:16.728766 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:16.729578 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:16.729619 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:16.732574 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:16.736683 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:16.736691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:16.736706 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:16.737159 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:16.737660 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:16.737688 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:16.737691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:16.737892 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:16.738691 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:16.738698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:16.739057 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 18:49:16.739442 (+ 385us) client_negotiation.cc:240] Negotiation successful
0620 18:49:16.739918 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":378,"threads_started":1}
I20260620 18:49:16.742746 2943 heartbeater.cc:347] Connected to a master server at 127.15.102.254:42987
I20260620 18:49:16.743077 2943 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:16.743892 2943 heartbeater.cc:511] Master 127.15.102.254:42987 requested a full tablet report, sending...
I20260620 18:49:16.745465 2618 ts_manager.cc:194] Registered new tserver with Master: 1e04555c6e41402b8b81cf1b0ed4d7d0 (127.15.102.194:33657)
I20260620 18:49:16.746295 2618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.194@KRBTEST.COM'} at 127.15.102.194:46963
WARNING: no policy specified for kudu/127.15.102.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
Entry for principal kudu/127.15.102.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab.
I20260620 18:49:16.778617 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:42987
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:17.111457 2948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:17.111763 2948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:17.111858 2948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:17.119192 2948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:17.119388 2948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:17.119524 2948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:49:17.130097 2948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.195.keytab
--principal=kudu/127.15.102.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:42987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:17.131462 2948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:17.132956 2948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:17.146276 2953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:17.146220 2954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:17.146220 2956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:17.147173 2948 server_base.cc:1034] running on GCE node
I20260620 18:49:17.147871 2948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:17.148865 2948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:17.150152 2948 hybrid_clock.cc:648] HybridClock initialized: now 1781981357150105 us; error 30 us; skew 500 ppm
I20260620 18:49:17.150436 2948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:17 dist-test-slave-zfp8 krb5kdc[2573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981357, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:17.154008 2948 init.cc:377] Logged in from keytab as kudu/127.15.102.195@KRBTEST.COM (short username kudu)
I20260620 18:49:17.155596 2948 webserver.cc:460] Webserver started at http://127.15.102.195:39185/ using document root <none> and password file <none>
I20260620 18:49:17.156296 2948 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:17.156399 2948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:17.156661 2948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:17.159843 2948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "858c7f102265440e8cf517ece4eab708"
format_stamp: "Formatted at 2026-06-20 18:49:17 on dist-test-slave-zfp8"
server_key: "d63b8af56c017ad33ee1f2f0e03e8288"
server_key_iv: "5db6309b47e41fc356938011496cccf2"
server_key_version: "encryptionkey@0"
I20260620 18:49:17.160570 2948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "858c7f102265440e8cf517ece4eab708"
format_stamp: "Formatted at 2026-06-20 18:49:17 on dist-test-slave-zfp8"
server_key: "d63b8af56c017ad33ee1f2f0e03e8288"
server_key_iv: "5db6309b47e41fc356938011496cccf2"
server_key_version: "encryptionkey@0"
I20260620 18:49:17.165759 2948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:49:17.169389 2963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:17.170789 2948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:49:17.170984 2948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "858c7f102265440e8cf517ece4eab708"
format_stamp: "Formatted at 2026-06-20 18:49:17 on dist-test-slave-zfp8"
server_key: "d63b8af56c017ad33ee1f2f0e03e8288"
server_key_iv: "5db6309b47e41fc356938011496cccf2"
server_key_version: "encryptionkey@0"
I20260620 18:49:17.171217 2948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:17.193881 2948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:17.198549 2948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:17.198871 2948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:17.199990 2948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:17.201397 2948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:17.201511 2948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:17.201668 2948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:17.201745 2948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:17.236238 2809 heartbeater.cc:503] Master 127.15.102.254:42987 was elected leader, sending a full tablet report...
I20260620 18:49:17.238428 2948 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:35481
I20260620 18:49:17.238530 3075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:35481 every 8 connection(s)
I20260620 18:49:17.240046 2948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:49:17.244947 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 2948
I20260620 18:49:17.245136 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:49:17.245559 15771 external_mini_cluster.cc:1428] Setting key fc11a0df462b50f914cbd8daca14a8a2
Jun 20 18:49:17 dist-test-slave-zfp8 krb5kdc[2573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981357, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.195@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:17.261260 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.242713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:33637 (local address 127.15.102.254:42987)
0620 18:49:17.243151 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.243156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.244240 (+ 1084us) server_negotiation.cc:407] Connection header received
0620 18:49:17.246220 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.246225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.246293 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.246388 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:17.248321 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.249248 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.250945 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.251496 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.251632 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.255054 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:17.255059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:17.255062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:17.255093 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:17.257661 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.258351 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.258355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.258357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.258474 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.258998 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.259004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.259007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.259414 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:17.259572 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.260674 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.260865 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
I20260620 18:49:17.262359 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.243349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:42987 (local address 127.15.102.195:33637)
0620 18:49:17.244073 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.244113 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:17.245934 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:17.246744 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:17.246759 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:17.247456 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:17.248152 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.248171 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.249403 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.249409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.250787 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.250799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.251743 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.251750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.251974 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.252588 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:17.252611 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:17.254873 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:17.257792 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:17.257799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:17.257812 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:17.258247 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:17.258597 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:17.258602 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:17.258605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:17.258878 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:17.259652 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:17.259658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:17.260257 (+ 599us) client_negotiation.cc:769] Sending connection context
0620 18:49:17.260542 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 18:49:17.261142 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":292,"threads_started":1}
I20260620 18:49:17.263912 3076 heartbeater.cc:347] Connected to a master server at 127.15.102.254:42987
I20260620 18:49:17.264195 3076 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:17.264894 3076 heartbeater.cc:511] Master 127.15.102.254:42987 requested a full tablet report, sending...
I20260620 18:49:17.266396 2618 ts_manager.cc:194] Registered new tserver with Master: 858c7f102265440e8cf517ece4eab708 (127.15.102.195:35481)
I20260620 18:49:17.267222 2618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.102.195@KRBTEST.COM'} at 127.15.102.195:33637
I20260620 18:49:17.273556 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:49:17.290477 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.277957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.15.102.254:42987)
0620 18:49:17.278148 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.278153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.278629 (+ 476us) server_negotiation.cc:407] Connection header received
0620 18:49:17.278768 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.278772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.278836 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.278925 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:17.280382 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.281112 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.282379 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.282761 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.282852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.285216 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:17.285221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:17.285225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:17.285259 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:17.288065 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.288770 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.288774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.288776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.288901 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.289237 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.289240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.289242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.289581 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:17.289768 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.290056 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.290234 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:49:17.294528 2618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:49:17.301654 2618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:49:17.327348 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.314057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:33657 (local address 127.0.0.1:41414)
0620 18:49:17.316447 (+ 2390us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.316513 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:17.316741 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:17.318663 (+ 1922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:17.318672 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:17.318707 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:17.319258 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.319283 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.321484 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.321492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.324113 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.324129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.326343 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.326351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.326537 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.326597 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 18:49:17.326715 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 18:49:17.326915 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2051,"thread_start_us":3912,"threads_started":1}
I20260620 18:49:17.328184 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.314319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.15.102.194:33657)
0620 18:49:17.317322 (+ 3003us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.317333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.317367 (+ 34us) server_negotiation.cc:407] Connection header received
0620 18:49:17.317478 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.317487 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.318383 (+ 896us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.319488 (+ 1105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:17.319713 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.321315 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.324326 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.325827 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.325983 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.326176 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.327686 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.327908 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2855,"thread_start_us":2756,"threads_started":1}
I20260620 18:49:17.328719 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.314943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:35481 (local address 127.0.0.1:36634)
0620 18:49:17.318256 (+ 3313us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.318280 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:17.318413 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:17.319092 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:17.319097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:17.319123 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:17.319636 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.319650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.322742 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.322749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.324889 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.324903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.327905 (+ 3002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.327911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.328083 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.328117 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:49:17.328231 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 18:49:17.328375 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3186,"thread_start_us":1125,"threads_started":1}
I20260620 18:49:17.328776 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.315738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.15.102.195:35481)
0620 18:49:17.317810 (+ 2072us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.317818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.318534 (+ 716us) server_negotiation.cc:407] Connection header received
0620 18:49:17.318639 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.318647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.318897 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.319105 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:17.319873 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.322582 (+ 2709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.325066 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.327645 (+ 2579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.327851 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.328026 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.328288 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.328453 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1877,"thread_start_us":1699,"threads_started":1}
I20260620 18:49:17.332669 2878 tablet_service.cc:1467] Processing CreateTablet for tablet ee196d8d8d254a4e9f29187a0211b2df (DEFAULT_TABLE table=test-table [id=8bab29e6bf4e47d7a5ee7f8a2fdd6652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:17.332927 3011 tablet_service.cc:1467] Processing CreateTablet for tablet ee196d8d8d254a4e9f29187a0211b2df (DEFAULT_TABLE table=test-table [id=8bab29e6bf4e47d7a5ee7f8a2fdd6652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:17.334144 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee196d8d8d254a4e9f29187a0211b2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:17.334379 3011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee196d8d8d254a4e9f29187a0211b2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:17.336042 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.319627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:43251 (local address 127.0.0.1:52240)
0620 18:49:17.327652 (+ 8025us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.327671 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:17.327791 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:17.328594 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:17.328601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:17.328632 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:17.329018 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.329030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.331206 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.331212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.333265 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.333285 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.335164 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.335173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.335407 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.335453 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 18:49:17.335559 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 18:49:17.335701 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7895,"thread_start_us":7750,"threads_started":1}
I20260620 18:49:17.336098 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.318596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.15.102.193:43251)
0620 18:49:17.327335 (+ 8739us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.327346 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.327929 (+ 583us) server_negotiation.cc:407] Connection header received
0620 18:49:17.328050 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.328059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.328364 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.329210 (+ 846us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:17.329446 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.331014 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.333457 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.334927 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.335134 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.335354 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.335627 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.335794 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8527,"thread_start_us":8368,"threads_started":1}
I20260620 18:49:17.339766 2743 tablet_service.cc:1467] Processing CreateTablet for tablet ee196d8d8d254a4e9f29187a0211b2df (DEFAULT_TABLE table=test-table [id=8bab29e6bf4e47d7a5ee7f8a2fdd6652]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:17.341173 2743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee196d8d8d254a4e9f29187a0211b2df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:17.348968 3095 tablet_bootstrap.cc:492] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Bootstrap starting.
I20260620 18:49:17.354339 3097 tablet_bootstrap.cc:492] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: Bootstrap starting.
I20260620 18:49:17.355674 3096 tablet_bootstrap.cc:492] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Bootstrap starting.
I20260620 18:49:17.356640 3095 tablet_bootstrap.cc:654] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:17.358074 3095 log.cc:824] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:17.358143 3097 tablet_bootstrap.cc:654] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:17.359302 3097 log.cc:824] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:17.359512 3096 tablet_bootstrap.cc:654] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:17.360852 3096 log.cc:824] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:17.362252 3095 tablet_bootstrap.cc:492] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: No bootstrap required, opened a new log
I20260620 18:49:17.362562 3095 ts_tablet_manager.cc:1397] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260620 18:49:17.363812 3097 tablet_bootstrap.cc:492] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: No bootstrap required, opened a new log
I20260620 18:49:17.363802 3096 tablet_bootstrap.cc:492] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: No bootstrap required, opened a new log
I20260620 18:49:17.364063 3096 ts_tablet_manager.cc:1397] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 18:49:17.364063 3097 ts_tablet_manager.cc:1397] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 18:49:17.370334 3095 raft_consensus.cc:348] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.370791 3095 raft_consensus.cc:374] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:17.370971 3095 raft_consensus.cc:729] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e04555c6e41402b8b81cf1b0ed4d7d0, State: Initialized, Role: FOLLOWER
I20260620 18:49:17.371681 3095 consensus_queue.cc:260] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.372890 2943 heartbeater.cc:503] Master 127.15.102.254:42987 was elected leader, sending a full tablet report...
I20260620 18:49:17.373505 3095 ts_tablet_manager.cc:1428] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 18:49:17.374787 3096 raft_consensus.cc:348] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.375444 3096 raft_consensus.cc:374] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:17.375579 3096 raft_consensus.cc:729] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858c7f102265440e8cf517ece4eab708, State: Initialized, Role: FOLLOWER
I20260620 18:49:17.375767 3097 raft_consensus.cc:348] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.376230 3097 raft_consensus.cc:374] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:17.376385 3097 raft_consensus.cc:729] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9268de3fe5e4be8b46e45e7766308d5, State: Initialized, Role: FOLLOWER
I20260620 18:49:17.376370 3096 consensus_queue.cc:260] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.377327 3097 consensus_queue.cc:260] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.379058 3076 heartbeater.cc:503] Master 127.15.102.254:42987 was elected leader, sending a full tablet report...
I20260620 18:49:17.379633 3096 ts_tablet_manager.cc:1428] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260620 18:49:17.382405 3097 ts_tablet_manager.cc:1428] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20260620 18:49:17.452764 2810 tablet.cc:2369] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:49:17.468152 2944 tablet.cc:2369] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:49:17.491852 3077 tablet.cc:2369] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:17.576371 3103 raft_consensus.cc:482] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:49:17.576777 3103 raft_consensus.cc:504] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.578579 3103 leader_election.cc:288] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e04555c6e41402b8b81cf1b0ed4d7d0 (127.15.102.194:33657), 858c7f102265440e8cf517ece4eab708 (127.15.102.195:35481)
I20260620 18:49:17.585500 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.579168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:40439 (local address 127.15.102.194:33657)
0620 18:49:17.579405 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.579410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.580027 (+ 617us) server_negotiation.cc:407] Connection header received
0620 18:49:17.580277 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.580282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.580354 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.580458 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:17.581044 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.581764 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.583601 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.584466 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.584621 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.584707 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.585073 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.585212 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:49:17.585572 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.579131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:33657 (local address 127.15.102.193:40439)
0620 18:49:17.579896 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.579912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:17.580115 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:17.580481 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:17.580486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:17.580514 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:17.580885 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.580895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.581909 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.581917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.583440 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.583452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.584690 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.584697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.584874 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.584922 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 18:49:17.585015 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 18:49:17.585167 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"spinlock_wait_cycles":24832,"thread_start_us":461,"threads_started":1}
I20260620 18:49:17.585883 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.579090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:47291 (local address 127.15.102.195:35481)
0620 18:49:17.579349 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.579358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.580537 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 18:49:17.580760 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.580765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.580823 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.580929 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:17.581468 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.582181 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.583955 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.584854 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.585027 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.585094 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.585470 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.585618 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:49:17.586161 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.579130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:35481 (local address 127.15.102.193:47291)
0620 18:49:17.580406 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.580420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:17.580599 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:17.580954 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:17.580958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:17.580983 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:17.581308 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.581316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.582339 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.582345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.583791 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:17.583803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.585107 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.585114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:17.585271 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.585309 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 18:49:17.585407 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 18:49:17.585546 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":242,"thread_start_us":664,"threads_started":1}
I20260620 18:49:17.586683 2898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee196d8d8d254a4e9f29187a0211b2df" candidate_uuid: "f9268de3fe5e4be8b46e45e7766308d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" is_pre_election: true
I20260620 18:49:17.587284 2898 raft_consensus.cc:2454] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9268de3fe5e4be8b46e45e7766308d5 in term 0.
I20260620 18:49:17.587293 3031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee196d8d8d254a4e9f29187a0211b2df" candidate_uuid: "f9268de3fe5e4be8b46e45e7766308d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858c7f102265440e8cf517ece4eab708" is_pre_election: true
I20260620 18:49:17.587852 3031 raft_consensus.cc:2454] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9268de3fe5e4be8b46e45e7766308d5 in term 0.
I20260620 18:49:17.588009 2697 leader_election.cc:302] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04555c6e41402b8b81cf1b0ed4d7d0, f9268de3fe5e4be8b46e45e7766308d5; no voters:
I20260620 18:49:17.588485 3103 raft_consensus.cc:2781] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:49:17.588642 3103 raft_consensus.cc:482] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:49:17.588748 3103 raft_consensus.cc:3037] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:17.590866 3103 raft_consensus.cc:504] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.591593 3103 leader_election.cc:288] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [CANDIDATE]: Term 1 election: Requested vote from peers 1e04555c6e41402b8b81cf1b0ed4d7d0 (127.15.102.194:33657), 858c7f102265440e8cf517ece4eab708 (127.15.102.195:35481)
I20260620 18:49:17.592049 2898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee196d8d8d254a4e9f29187a0211b2df" candidate_uuid: "f9268de3fe5e4be8b46e45e7766308d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0"
I20260620 18:49:17.592049 3031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee196d8d8d254a4e9f29187a0211b2df" candidate_uuid: "f9268de3fe5e4be8b46e45e7766308d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858c7f102265440e8cf517ece4eab708"
I20260620 18:49:17.592278 3031 raft_consensus.cc:3037] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:17.592271 2898 raft_consensus.cc:3037] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:17.595552 3031 raft_consensus.cc:2454] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9268de3fe5e4be8b46e45e7766308d5 in term 1.
I20260620 18:49:17.595552 2898 raft_consensus.cc:2454] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9268de3fe5e4be8b46e45e7766308d5 in term 1.
I20260620 18:49:17.595970 2697 leader_election.cc:302] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e04555c6e41402b8b81cf1b0ed4d7d0, f9268de3fe5e4be8b46e45e7766308d5; no voters:
I20260620 18:49:17.596297 3103 raft_consensus.cc:2781] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:17.596756 3103 raft_consensus.cc:686] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 1 LEADER]: Becoming Leader. State: Replica: f9268de3fe5e4be8b46e45e7766308d5, State: Running, Role: LEADER
I20260620 18:49:17.597271 3103 consensus_queue.cc:237] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } }
I20260620 18:49:17.602150 2618 catalog_manager.cc:5495] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9268de3fe5e4be8b46e45e7766308d5 (127.15.102.193). New cstate: current_term: 1 leader_uuid: "f9268de3fe5e4be8b46e45e7766308d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9268de3fe5e4be8b46e45e7766308d5" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 43251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 } health_report { overall_health: UNKNOWN } } }
I20260620 18:49:17.621443 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.615825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.15.102.193:43251)
0620 18:49:17.616066 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.616072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.616185 (+ 113us) server_negotiation.cc:407] Connection header received
0620 18:49:17.616455 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.616459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.616565 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.616732 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:17.617125 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.617832 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.619214 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.619670 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.619818 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.620317 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:17.620849 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:17.621029 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.621082 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.621168 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:49:17.629606 3031 raft_consensus.cc:1261] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 1 FOLLOWER]: Refusing update from remote peer f9268de3fe5e4be8b46e45e7766308d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:17.629663 2898 raft_consensus.cc:1261] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 1 FOLLOWER]: Refusing update from remote peer f9268de3fe5e4be8b46e45e7766308d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:17.630650 3106 consensus_queue.cc:1035] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858c7f102265440e8cf517ece4eab708" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 35481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:17.631040 3103 consensus_queue.cc:1035] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e04555c6e41402b8b81cf1b0ed4d7d0" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 33657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:17.656313 2618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:17.656631 2618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:17.660543 2618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f122b0327d24e75a90adaea03eda733: Sending DeleteTablet for 3 replicas of tablet ee196d8d8d254a4e9f29187a0211b2df
I20260620 18:49:17.661955 3011 tablet_service.cc:1514] Processing DeleteTablet for tablet ee196d8d8d254a4e9f29187a0211b2df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:17 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:36634
I20260620 18:49:17.662299 2743 tablet_service.cc:1514] Processing DeleteTablet for tablet ee196d8d8d254a4e9f29187a0211b2df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:17 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:52240
I20260620 18:49:17.662716 2878 tablet_service.cc:1514] Processing DeleteTablet for tablet ee196d8d8d254a4e9f29187a0211b2df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:17 UTC) from {username='kudu', principal='kudu/127.15.102.254@KRBTEST.COM'} at 127.0.0.1:41414
I20260620 18:49:17.663144 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 2680
I20260620 18:49:17.678715 3120 tablet_replica.cc:330] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: stopping tablet replica
I20260620 18:49:17.679402 3120 raft_consensus.cc:2229] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:17.680088 3120 raft_consensus.cc:2258] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:17.681049 3118 tablet_replica.cc:330] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: stopping tablet replica
I20260620 18:49:17.681584 3118 raft_consensus.cc:2229] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:17.682263 3118 raft_consensus.cc:2258] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:17.684772 3120 ts_tablet_manager.cc:1905] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 18:49:17 dist-test-slave-zfp8 krb5kdc[2573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.193@KRBTEST.COM
I20260620 18:49:17.691805 3119 tablet_replica.cc:330] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: stopping tablet replica
I20260620 18:49:17.692245 3119 raft_consensus.cc:2229] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:49:17.692994 3119 raft_consensus.cc:2258] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:17.693198 3120 ts_tablet_manager.cc:1918] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:17.695154 3118 ts_tablet_manager.cc:1905] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:17.695322 3119 ts_tablet_manager.cc:1905] T ee196d8d8d254a4e9f29187a0211b2df P f9268de3fe5e4be8b46e45e7766308d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:17.696323 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.664289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.15.102.193:43251)
0620 18:49:17.664476 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.664482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.675140 (+ 10658us) server_negotiation.cc:407] Connection header received
0620 18:49:17.675477 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.675486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.675593 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.675693 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:17.677898 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.679558 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.681480 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.682240 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.682378 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.688146 (+ 5768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:17.688166 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:17.688190 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:17.688245 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:17.692556 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.693503 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.693515 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.693522 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.693691 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.694188 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.694193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.694196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.694687 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:17.694991 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.695666 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.695954 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:49:17.698132 2804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:17.703490 3120 log.cc:1196] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/ee196d8d8d254a4e9f29187a0211b2df
I20260620 18:49:17.704046 3120 ts_tablet_manager.cc:1939] T ee196d8d8d254a4e9f29187a0211b2df P 1e04555c6e41402b8b81cf1b0ed4d7d0: Deleting consensus metadata
I20260620 18:49:17.705879 2603 catalog_manager.cc:4841] TS 1e04555c6e41402b8b81cf1b0ed4d7d0 (127.15.102.194:33657): tablet ee196d8d8d254a4e9f29187a0211b2df (table test-table [id=8bab29e6bf4e47d7a5ee7f8a2fdd6652]) successfully deleted
I20260620 18:49:17.707429 3118 ts_tablet_manager.cc:1918] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:17.707607 3118 log.cc:1196] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/ee196d8d8d254a4e9f29187a0211b2df
I20260620 18:49:17.708138 3118 ts_tablet_manager.cc:1939] T ee196d8d8d254a4e9f29187a0211b2df P 858c7f102265440e8cf517ece4eab708: Deleting consensus metadata
I20260620 18:49:17.710120 2604 catalog_manager.cc:4841] TS 858c7f102265440e8cf517ece4eab708 (127.15.102.195:35481): tablet ee196d8d8d254a4e9f29187a0211b2df (table test-table [id=8bab29e6bf4e47d7a5ee7f8a2fdd6652]) successfully deleted
I20260620 18:49:17.814595 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 2680
W20260620 18:49:17.832954 2603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:43251 (error 108)
I20260620 18:49:17.833680 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 2815
I20260620 18:49:17.834919 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.834384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:43251 (local address 127.0.0.1:52274)
0620 18:49:17.834566 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.834686 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 18:49:17.835182 2603 catalog_manager.cc:4584] TS f9268de3fe5e4be8b46e45e7766308d5 (127.15.102.193:43251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee196d8d8d254a4e9f29187a0211b2df: Network error: Client connection negotiation failed: client connection to 127.15.102.193:43251: connect: Connection refused (error 111)
Jun 20 18:49:17 dist-test-slave-zfp8 krb5kdc[2573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.194@KRBTEST.COM
I20260620 18:49:17.848688 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.834506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.15.102.194:33657)
0620 18:49:17.834687 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.834715 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.834840 (+ 125us) server_negotiation.cc:407] Connection header received
0620 18:49:17.835032 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.835037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.835142 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.835294 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:17.836640 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.837379 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.838651 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.839027 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.839123 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.843503 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:17.843518 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:17.843534 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:17.843571 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:17.846181 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.846857 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.846865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.846867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.846992 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.847390 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.847393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.847395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.847775 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:17.848007 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.848279 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.848442 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:49:17.849373 2938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:17.888971 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.888468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:43251 (local address 127.0.0.1:52280)
0620 18:49:17.888698 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.888800 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 18:49:17.890023 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.889653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:43251 (local address 127.0.0.1:52284)
0620 18:49:17.889830 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.889900 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 18:49:17.941148 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 2815
I20260620 18:49:17.950484 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.949998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:43251 (local address 127.0.0.1:52300)
0620 18:49:17.950217 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.950313 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:49:17.951611 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.951280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:43251 (local address 127.0.0.1:52310)
0620 18:49:17.951425 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:17.951503 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 18:49:17.958935 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 2948
Jun 20 18:49:17 dist-test-slave-zfp8 krb5kdc[2573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.195@KRBTEST.COM
I20260620 18:49:17.974114 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:17.959736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.15.102.195:35481)
0620 18:49:17.959916 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:17.959921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:17.959940 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:49:17.960069 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:17.960073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:17.960140 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:17.960238 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:17.961637 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.962347 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.963734 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:17.964146 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:17.964244 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:17.968750 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:17.968763 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:17.968780 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:17.968813 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:17.971533 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.972282 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.972289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.972292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.972472 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:17.972852 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:17.972855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:17.972857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:17.973198 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:17.973433 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:17.973680 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 18:49:17.973852 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:49:17.974864 3071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:18.033242 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:18.032655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:43251 (local address 127.0.0.1:52320)
0620 18:49:18.032944 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:18.033048 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260620 18:49:18.034427 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:18.034049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:43251 (local address 127.0.0.1:52330)
0620 18:49:18.034246 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:18.034321 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:43251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 18:49:18.068203 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 2948
I20260620 18:49:18.085791 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 2586
I20260620 18:49:18.086982 2648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:18.199914 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 2586
2026-06-20T18:49:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4421 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T18:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:18Z Disabled control of system clock
I20260620 18:49:18.277907 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:43791
--webserver_interface=127.15.102.254
--webserver_port=0
--builtin_ntp_servers=127.15.102.212:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:49:18.624599 3130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:18.624910 3130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:18.625000 3130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:18.632277 3130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:18.632432 3130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:18.632478 3130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:18.632545 3130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:18.632591 3130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:18.641906 3130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:43791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:18.643268 3130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:18.645087 3130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:18.657675 3135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:18.657590 3136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:18.657724 3138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:18.658906 3130 server_base.cc:1034] running on GCE node
I20260620 18:49:18.659788 3130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:18.661298 3130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:18.662504 3130 hybrid_clock.cc:648] HybridClock initialized: now 1781981358662453 us; error 46 us; skew 500 ppm
I20260620 18:49:18.662923 3130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:18.665266 3130 webserver.cc:460] Webserver started at http://127.15.102.254:42747/ using document root <none> and password file <none>
I20260620 18:49:18.665987 3130 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:18.666106 3130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:18.666429 3130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:18.669466 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "906bcf3816a642f594589df4fe905f2e"
format_stamp: "Formatted at 2026-06-20 18:49:18 on dist-test-slave-zfp8"
server_key: "da8baab6bfc5de325e750effe185eea7"
server_key_iv: "41d2f535786d85c0676d4e270bfc419d"
server_key_version: "encryptionkey@0"
I20260620 18:49:18.670322 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "906bcf3816a642f594589df4fe905f2e"
format_stamp: "Formatted at 2026-06-20 18:49:18 on dist-test-slave-zfp8"
server_key: "da8baab6bfc5de325e750effe185eea7"
server_key_iv: "41d2f535786d85c0676d4e270bfc419d"
server_key_version: "encryptionkey@0"
I20260620 18:49:18.675627 3130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:49:18.679076 3144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:18.680546 3130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:18.680757 3130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "906bcf3816a642f594589df4fe905f2e"
format_stamp: "Formatted at 2026-06-20 18:49:18 on dist-test-slave-zfp8"
server_key: "da8baab6bfc5de325e750effe185eea7"
server_key_iv: "41d2f535786d85c0676d4e270bfc419d"
server_key_version: "encryptionkey@0"
I20260620 18:49:18.680984 3130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:18.699744 3130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:18.700603 3130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:18.700892 3130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:18.719529 3130 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:43791
I20260620 18:49:18.719553 3195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:43791 every 8 connection(s)
I20260620 18:49:18.720894 3130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:18.722481 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 3130
I20260620 18:49:18.722705 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:18.723167 15771 external_mini_cluster.cc:1428] Setting key f0a1809c95eff418745f24d5cbafc48d
I20260620 18:49:18.725245 3196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:18.738718 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:18.725136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.15.102.254:43791)
0620 18:49:18.726329 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:18.726343 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:18.726394 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:49:18.728500 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:18.728527 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:18.729136 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:18.729597 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:18.730138 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:18.732226 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:18.733524 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:18.734530 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:18.734710 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:18.735858 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:18.735869 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:18.735888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:18.736660 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:18.736757 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:18.736783 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:18.736916 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:18.737079 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 18:49:18.737441 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":439,"threads_started":1}
I20260620 18:49:18.740182 3196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e: Bootstrap starting.
I20260620 18:49:18.743172 3196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:18.744184 3196 log.cc:824] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:18.746809 3196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e: No bootstrap required, opened a new log
I20260620 18:49:18.753897 3196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906bcf3816a642f594589df4fe905f2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43791 } }
I20260620 18:49:18.754256 3196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:18.754364 3196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906bcf3816a642f594589df4fe905f2e, State: Initialized, Role: FOLLOWER
I20260620 18:49:18.754920 3196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906bcf3816a642f594589df4fe905f2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43791 } }
I20260620 18:49:18.755165 3196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:18.755332 3196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:18.755512 3196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:18.757606 3196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906bcf3816a642f594589df4fe905f2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43791 } }
I20260620 18:49:18.758067 3196 leader_election.cc:302] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 906bcf3816a642f594589df4fe905f2e; no voters:
I20260620 18:49:18.758591 3196 leader_election.cc:288] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:18.758777 3201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:18.759511 3201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [term 1 LEADER]: Becoming Leader. State: Replica: 906bcf3816a642f594589df4fe905f2e, State: Running, Role: LEADER
I20260620 18:49:18.760010 3201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906bcf3816a642f594589df4fe905f2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43791 } }
I20260620 18:49:18.760792 3196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:18.762637 3203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 906bcf3816a642f594589df4fe905f2e. Latest consensus state: current_term: 1 leader_uuid: "906bcf3816a642f594589df4fe905f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906bcf3816a642f594589df4fe905f2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43791 } } }
I20260620 18:49:18.763010 3203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:18.764135 3202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "906bcf3816a642f594589df4fe905f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "906bcf3816a642f594589df4fe905f2e" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43791 } } }
I20260620 18:49:18.764582 3202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:18.766824 3206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:18.772698 3206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:18.781901 3206 catalog_manager.cc:1347] Generated new cluster ID: 89d42fc4b8084a81881dc9900d650336
I20260620 18:49:18.782027 3206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:18.799826 3206 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:18.801156 3206 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:18.808787 3206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e: Generated new TSK 0
I20260620 18:49:18.809639 3206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:18.821408 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.193:0
--local_ip_for_outbound_sockets=127.15.102.193
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:43791
--builtin_ntp_servers=127.15.102.212:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 18:49:19.179358 3220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:19.179675 3220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:19.179870 3220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:19.187422 3220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:19.187526 3220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:19.187700 3220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.193
I20260620 18:49:19.196903 3220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.102.193
--webserver_port=0
--tserver_master_addrs=127.15.102.254:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.193
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:19.198267 3220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:19.200196 3220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:19.214015 3228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:19.214128 3226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:19.213981 3225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:19.220177 3220 server_base.cc:1034] running on GCE node
I20260620 18:49:19.220886 3220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:19.221982 3220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:19.223217 3220 hybrid_clock.cc:648] HybridClock initialized: now 1781981359223177 us; error 32 us; skew 500 ppm
I20260620 18:49:19.223641 3220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:19.225978 3220 webserver.cc:460] Webserver started at http://127.15.102.193:39333/ using document root <none> and password file <none>
I20260620 18:49:19.226718 3220 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:19.226854 3220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:19.227196 3220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:19.230223 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/data/instance:
uuid: "8ba3379278734825acdc38ad79f3c3a1"
format_stamp: "Formatted at 2026-06-20 18:49:19 on dist-test-slave-zfp8"
server_key: "6afdceaa7db19528f30f6588abababbe"
server_key_iv: "5fc763409c504dccbd90a6948166fce7"
server_key_version: "encryptionkey@0"
I20260620 18:49:19.231072 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance:
uuid: "8ba3379278734825acdc38ad79f3c3a1"
format_stamp: "Formatted at 2026-06-20 18:49:19 on dist-test-slave-zfp8"
server_key: "6afdceaa7db19528f30f6588abababbe"
server_key_iv: "5fc763409c504dccbd90a6948166fce7"
server_key_version: "encryptionkey@0"
I20260620 18:49:19.236229 3220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 18:49:19.240470 3234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:19.241829 3220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:49:19.242010 3220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/wal
uuid: "8ba3379278734825acdc38ad79f3c3a1"
format_stamp: "Formatted at 2026-06-20 18:49:19 on dist-test-slave-zfp8"
server_key: "6afdceaa7db19528f30f6588abababbe"
server_key_iv: "5fc763409c504dccbd90a6948166fce7"
server_key_version: "encryptionkey@0"
I20260620 18:49:19.242234 3220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:19.265650 3220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:19.266569 3220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:19.266894 3220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:19.267982 3220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:19.269457 3220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:19.269584 3220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:19.269716 3220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:19.269816 3220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:19.303500 3220 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.193:34437
I20260620 18:49:19.303608 3346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.193:34437 every 8 connection(s)
I20260620 18:49:19.304915 3220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/data/info.pb
I20260620 18:49:19.312000 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 3220
I20260620 18:49:19.312194 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-0/wal/instance
I20260620 18:49:19.312649 15771 external_mini_cluster.cc:1428] Setting key 40d7e480579bbf02d9254fa281818194
I20260620 18:49:19.315419 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.194:0
--local_ip_for_outbound_sockets=127.15.102.194
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:43791
--builtin_ntp_servers=127.15.102.212:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:49:19.319303 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:19.307721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.193:45831 (local address 127.15.102.254:43791)
0620 18:49:19.308170 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:19.308175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:19.309510 (+ 1335us) server_negotiation.cc:407] Connection header received
0620 18:49:19.311506 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:19.311512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:19.311609 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:19.311735 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:19.313434 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:19.314279 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:19.316191 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:19.316621 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:19.316781 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:19.318294 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:19.318300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:19.318303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:19.318323 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:19.318368 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:19.318381 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:19.318493 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:19.318743 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 18:49:19.318851 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":339,"thread_start_us":241,"threads_started":1}
I20260620 18:49:19.320312 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:19.308304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:43791 (local address 127.15.102.193:45831)
0620 18:49:19.309345 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:19.309383 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:19.311156 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:19.311970 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:19.311980 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:19.312089 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:19.313221 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:19.313242 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:19.314498 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:19.314510 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:19.316010 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:19.316023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:19.316916 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:19.316925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:19.317182 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:19.317930 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:19.317933 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:19.317981 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:19.318073 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:19.318491 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:19.318504 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:19.318515 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 18:49:19.318732 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 18:49:19.319142 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":271,"threads_started":1}
I20260620 18:49:19.321986 3347 heartbeater.cc:347] Connected to a master server at 127.15.102.254:43791
I20260620 18:49:19.322322 3347 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:19.323089 3347 heartbeater.cc:511] Master 127.15.102.254:43791 requested a full tablet report, sending...
I20260620 18:49:19.325428 3161 ts_manager.cc:194] Registered new tserver with Master: 8ba3379278734825acdc38ad79f3c3a1 (127.15.102.193:34437)
I20260620 18:49:19.327713 3161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.193:45831
W20260620 18:49:19.651103 3351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:19.651464 3351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:19.651554 3351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:19.658915 3351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:19.659013 3351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:19.659119 3351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.194
I20260620 18:49:19.668785 3351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.102.194
--webserver_port=0
--tserver_master_addrs=127.15.102.254:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.194
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:19.670037 3351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:19.671922 3351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:19.685303 3356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:19.685518 3357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:19.685667 3359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:19.686357 3351 server_base.cc:1034] running on GCE node
I20260620 18:49:19.686980 3351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:19.688020 3351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:19.689260 3351 hybrid_clock.cc:648] HybridClock initialized: now 1781981359689186 us; error 93 us; skew 500 ppm
I20260620 18:49:19.689538 3351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:19.691861 3351 webserver.cc:460] Webserver started at http://127.15.102.194:36355/ using document root <none> and password file <none>
I20260620 18:49:19.692541 3351 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:19.692628 3351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:19.692891 3351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:19.695801 3351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/data/instance:
uuid: "3558df1427f94598a39b5bc6a0d2cdca"
format_stamp: "Formatted at 2026-06-20 18:49:19 on dist-test-slave-zfp8"
server_key: "3d79933502e5e0bd453cc53939694cc1"
server_key_iv: "3dee9b7de9189f04f6519596b765cb96"
server_key_version: "encryptionkey@0"
I20260620 18:49:19.696539 3351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance:
uuid: "3558df1427f94598a39b5bc6a0d2cdca"
format_stamp: "Formatted at 2026-06-20 18:49:19 on dist-test-slave-zfp8"
server_key: "3d79933502e5e0bd453cc53939694cc1"
server_key_iv: "3dee9b7de9189f04f6519596b765cb96"
server_key_version: "encryptionkey@0"
I20260620 18:49:19.701603 3351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 18:49:19.705191 3365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:19.706594 3351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:49:19.706804 3351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/wal
uuid: "3558df1427f94598a39b5bc6a0d2cdca"
format_stamp: "Formatted at 2026-06-20 18:49:19 on dist-test-slave-zfp8"
server_key: "3d79933502e5e0bd453cc53939694cc1"
server_key_iv: "3dee9b7de9189f04f6519596b765cb96"
server_key_version: "encryptionkey@0"
I20260620 18:49:19.707039 3351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:19.721908 3351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:19.722865 3351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:19.723160 3351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:19.724268 3351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:19.726074 3351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:19.726182 3351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:19.726289 3351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:19.726363 3351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:19.764322 3351 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.194:45575
I20260620 18:49:19.764431 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.194:45575 every 8 connection(s)
I20260620 18:49:19.765802 3351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/data/info.pb
I20260620 18:49:19.773579 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 3351
I20260620 18:49:19.773785 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/wal/instance
I20260620 18:49:19.774220 15771 external_mini_cluster.cc:1428] Setting key 1753b91f28cfca976f16ef13134366eb
I20260620 18:49:19.777331 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.102.195:0
--local_ip_for_outbound_sockets=127.15.102.195
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:43791
--builtin_ntp_servers=127.15.102.212:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 18:49:19.779454 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:19.768591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.194:33977 (local address 127.15.102.254:43791)
0620 18:49:19.768793 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:19.768797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:19.770142 (+ 1345us) server_negotiation.cc:407] Connection header received
0620 18:49:19.772072 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:19.772077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:19.772197 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:19.772347 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:19.773554 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:19.774457 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:19.776483 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:19.776886 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:19.776986 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:19.778617 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:19.778622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:19.778625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:19.778643 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:19.778675 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:19.778687 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:19.778762 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:19.779033 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:49:19.779116 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:49:19.780946 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:19.768954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:43791 (local address 127.15.102.194:33977)
0620 18:49:19.769947 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:19.769985 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:19.771789 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:19.772585 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:19.772594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:19.772691 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:19.773355 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:19.773382 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:19.774627 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:19.774635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:19.776335 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:19.776350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:19.777127 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:19.777138 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:19.777444 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:19.778228 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:19.778234 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:19.778292 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:19.778405 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:19.778795 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:19.778808 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:19.778820 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 18:49:19.779015 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 18:49:19.779490 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":365,"threads_started":1}
I20260620 18:49:19.782464 3479 heartbeater.cc:347] Connected to a master server at 127.15.102.254:43791
I20260620 18:49:19.782788 3479 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:19.783650 3479 heartbeater.cc:511] Master 127.15.102.254:43791 requested a full tablet report, sending...
I20260620 18:49:19.785293 3161 ts_manager.cc:194] Registered new tserver with Master: 3558df1427f94598a39b5bc6a0d2cdca (127.15.102.194:45575)
I20260620 18:49:19.786255 3161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.194:33977
W20260620 18:49:20.115026 3482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:20.115358 3482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:20.115526 3482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:20.122932 3482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:20.123032 3482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:20.123234 3482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.102.195
I20260620 18:49:20.133157 3482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.102.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.102.195
--webserver_port=0
--tserver_master_addrs=127.15.102.254:43791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.102.195
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:20.134474 3482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:20.136363 3482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:20.149586 3487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:20.149765 3488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:20.149725 3490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:20.150949 3482 server_base.cc:1034] running on GCE node
I20260620 18:49:20.151607 3482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:20.152644 3482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:20.153882 3482 hybrid_clock.cc:648] HybridClock initialized: now 1781981360153819 us; error 52 us; skew 500 ppm
I20260620 18:49:20.154199 3482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 18:49:20.156428 3482 webserver.cc:460] Webserver started at http://127.15.102.195:36669/ using document root <none> and password file <none>
I20260620 18:49:20.157128 3482 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:20.157261 3482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:20.157579 3482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:20.160820 3482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/data/instance:
uuid: "ff992e7c81474a15bf7d20b241382646"
format_stamp: "Formatted at 2026-06-20 18:49:20 on dist-test-slave-zfp8"
server_key: "bbcc0c53f6bcbf18a179eaa3c464f75a"
server_key_iv: "b2bd9ce7eb650c8429ecf4d2c42b1101"
server_key_version: "encryptionkey@0"
I20260620 18:49:20.161724 3482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance:
uuid: "ff992e7c81474a15bf7d20b241382646"
format_stamp: "Formatted at 2026-06-20 18:49:20 on dist-test-slave-zfp8"
server_key: "bbcc0c53f6bcbf18a179eaa3c464f75a"
server_key_iv: "b2bd9ce7eb650c8429ecf4d2c42b1101"
server_key_version: "encryptionkey@0"
I20260620 18:49:20.167340 3482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:49:20.171437 3496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:20.172797 3482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:20.172986 3482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/wal
uuid: "ff992e7c81474a15bf7d20b241382646"
format_stamp: "Formatted at 2026-06-20 18:49:20 on dist-test-slave-zfp8"
server_key: "bbcc0c53f6bcbf18a179eaa3c464f75a"
server_key_iv: "b2bd9ce7eb650c8429ecf4d2c42b1101"
server_key_version: "encryptionkey@0"
I20260620 18:49:20.173244 3482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:20.195267 3482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:20.196164 3482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:20.196491 3482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:20.197505 3482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 18:49:20.198999 3482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 18:49:20.199129 3482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:20.199285 3482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 18:49:20.199385 3482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 18:49:20.232949 3482 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.195:42327
I20260620 18:49:20.233044 3608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.195:42327 every 8 connection(s)
I20260620 18:49:20.234273 3482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/data/info.pb
I20260620 18:49:20.244570 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 3482
I20260620 18:49:20.244777 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/wal/instance
I20260620 18:49:20.245273 15771 external_mini_cluster.cc:1428] Setting key 91e62679dc9695328b53c089ee4edd70
I20260620 18:49:20.248034 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.236991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:33121 (local address 127.15.102.254:43791)
0620 18:49:20.237229 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.237233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.238490 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 18:49:20.240524 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.240531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.240631 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.240766 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:20.241906 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.242715 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.244770 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.245257 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.245432 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.247096 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:20.247104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:20.247110 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:20.247138 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:20.247186 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:20.247205 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:20.247343 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.247554 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.247680 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:49:20.249053 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.237361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:43791 (local address 127.15.102.195:33121)
0620 18:49:20.238300 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.238339 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:20.240214 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:20.241012 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:20.241021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:20.241112 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:20.241712 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.241731 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.242899 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.242911 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.244582 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.244600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.245510 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.245517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.245818 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.246738 (+ 920us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 18:49:20.246741 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:20.246789 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 18:49:20.246872 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:20.247337 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:20.247345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:20.247351 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 18:49:20.247539 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 18:49:20.247942 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":281,"threads_started":1}
I20260620 18:49:20.250411 3609 heartbeater.cc:347] Connected to a master server at 127.15.102.254:43791
I20260620 18:49:20.250707 3609 heartbeater.cc:464] Registering TS with master...
I20260620 18:49:20.251513 3609 heartbeater.cc:511] Master 127.15.102.254:43791 requested a full tablet report, sending...
I20260620 18:49:20.252848 3161 ts_manager.cc:194] Registered new tserver with Master: ff992e7c81474a15bf7d20b241382646 (127.15.102.195:42327)
I20260620 18:49:20.253649 3161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.102.195:33121
I20260620 18:49:20.260691 15771 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 18:49:20.270955 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.264778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.15.102.254:43791)
0620 18:49:20.264970 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.264975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.265510 (+ 535us) server_negotiation.cc:407] Connection header received
0620 18:49:20.265648 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.265652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.265726 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.265810 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:20.266384 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.267140 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.268734 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.269138 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.269243 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.270353 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:20.270358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:20.270361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:20.270380 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:20.270409 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:20.270421 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:20.270500 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.270635 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.270711 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:49:20.275314 3161 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 18:49:20.282495 3161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 18:49:20.307520 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.295054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:45575 (local address 127.0.0.1:34610)
0620 18:49:20.298138 (+ 3084us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.298155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:20.298264 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:20.299629 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:20.299634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:20.299660 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:20.300024 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.300036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.301730 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.301735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.304260 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.304276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.306457 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.306463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.306653 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.306704 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 18:49:20.306831 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 18:49:20.307142 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2958,"mutex_wait_us":53,"thread_start_us":3824,"threads_started":1}
I20260620 18:49:20.307861 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.295284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.15.102.195:42327)
0620 18:49:20.296175 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.296186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.297371 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 18:49:20.297778 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.297790 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.298082 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.298274 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:20.299392 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.300963 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.303845 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.305272 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.306531 (+ 1259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.306766 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.307335 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.307505 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":513,"threads_started":1}
I20260620 18:49:20.308561 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.294980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.195:42327 (local address 127.0.0.1:50858)
0620 18:49:20.297228 (+ 2248us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.297265 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:20.297534 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:20.298621 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:20.298628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:20.298665 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:20.299145 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.299161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.301102 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.301109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.303624 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.303641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.306898 (+ 3257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.306904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.307105 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.307135 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 18:49:20.308112 (+ 977us) client_negotiation.cc:240] Negotiation successful
0620 18:49:20.308254 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1992,"mutex_wait_us":86,"thread_start_us":2885,"threads_started":1}
I20260620 18:49:20.309065 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.295727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.15.102.193:34437)
0620 18:49:20.296589 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.296597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.299332 (+ 2735us) server_negotiation.cc:407] Connection header received
0620 18:49:20.299625 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.299636 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.299863 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.300028 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:20.300771 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.302191 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.306082 (+ 3891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.307574 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.307759 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.307965 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.308561 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.308702 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":814,"threads_started":1}
I20260620 18:49:20.309083 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.295525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.15.102.194:45575)
0620 18:49:20.298738 (+ 3213us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.298755 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.298787 (+ 32us) server_negotiation.cc:407] Connection header received
0620 18:49:20.298891 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.298897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.299114 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.299386 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:20.300205 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.301576 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.304494 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.305895 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.306092 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.306320 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.308449 (+ 2129us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.308624 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3057,"thread_start_us":2941,"threads_started":1}
I20260620 18:49:20.309991 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.296360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:56582)
0620 18:49:20.299150 (+ 2790us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.299166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:20.299433 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:20.300227 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:20.300232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:20.300255 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:20.300583 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.300594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.302355 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.302360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.305868 (+ 3508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.305884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.308146 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.308152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.308340 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.308374 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 18:49:20.309575 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0620 18:49:20.309704 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2667,"thread_start_us":475,"threads_started":1}
I20260620 18:49:20.312711 3544 tablet_service.cc:1467] Processing CreateTablet for tablet a380afdf995b44f8908f9f5107f20636 (DEFAULT_TABLE table=test-table [id=c3c9c224398f45899b264338ef4ebe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:20.312772 3413 tablet_service.cc:1467] Processing CreateTablet for tablet a380afdf995b44f8908f9f5107f20636 (DEFAULT_TABLE table=test-table [id=c3c9c224398f45899b264338ef4ebe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:20.313557 3282 tablet_service.cc:1467] Processing CreateTablet for tablet a380afdf995b44f8908f9f5107f20636 (DEFAULT_TABLE table=test-table [id=c3c9c224398f45899b264338ef4ebe4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 18:49:20.314054 3544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a380afdf995b44f8908f9f5107f20636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:20.314190 3413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a380afdf995b44f8908f9f5107f20636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:20.314802 3282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a380afdf995b44f8908f9f5107f20636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:20.325289 3627 tablet_bootstrap.cc:492] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Bootstrap starting.
I20260620 18:49:20.328742 3627 tablet_bootstrap.cc:654] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:20.329846 3628 tablet_bootstrap.cc:492] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1: Bootstrap starting.
I20260620 18:49:20.330107 3627 log.cc:824] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:20.331132 3347 heartbeater.cc:503] Master 127.15.102.254:43791 was elected leader, sending a full tablet report...
I20260620 18:49:20.334560 3629 tablet_bootstrap.cc:492] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: Bootstrap starting.
I20260620 18:49:20.334932 3628 tablet_bootstrap.cc:654] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:20.336551 3628 log.cc:824] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:20.339023 3627 tablet_bootstrap.cc:492] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: No bootstrap required, opened a new log
I20260620 18:49:20.339329 3627 ts_tablet_manager.cc:1397] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s
I20260620 18:49:20.339479 3629 tablet_bootstrap.cc:654] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:20.340137 3628 tablet_bootstrap.cc:492] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1: No bootstrap required, opened a new log
I20260620 18:49:20.340413 3628 ts_tablet_manager.cc:1397] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260620 18:49:20.340660 3629 log.cc:824] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:20.343667 3629 tablet_bootstrap.cc:492] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: No bootstrap required, opened a new log
I20260620 18:49:20.343974 3629 ts_tablet_manager.cc:1397] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260620 18:49:20.348941 3628 raft_consensus.cc:348] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.349332 3628 raft_consensus.cc:374] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:20.349431 3628 raft_consensus.cc:729] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba3379278734825acdc38ad79f3c3a1, State: Initialized, Role: FOLLOWER
I20260620 18:49:20.350057 3628 consensus_queue.cc:260] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.349992 3627 raft_consensus.cc:348] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.350430 3627 raft_consensus.cc:374] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:20.350590 3627 raft_consensus.cc:729] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff992e7c81474a15bf7d20b241382646, State: Initialized, Role: FOLLOWER
I20260620 18:49:20.351265 3627 consensus_queue.cc:260] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.352094 3628 ts_tablet_manager.cc:1428] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260620 18:49:20.353139 3609 heartbeater.cc:503] Master 127.15.102.254:43791 was elected leader, sending a full tablet report...
I20260620 18:49:20.353763 3627 ts_tablet_manager.cc:1428] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260620 18:49:20.354676 3629 raft_consensus.cc:348] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.355289 3629 raft_consensus.cc:374] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:20.355406 3629 raft_consensus.cc:729] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3558df1427f94598a39b5bc6a0d2cdca, State: Initialized, Role: FOLLOWER
I20260620 18:49:20.356043 3629 consensus_queue.cc:260] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.357168 3479 heartbeater.cc:503] Master 127.15.102.254:43791 was elected leader, sending a full tablet report...
I20260620 18:49:20.357760 3629 ts_tablet_manager.cc:1428] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260620 18:49:20.415126 3634 raft_consensus.cc:482] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 18:49:20.415534 3634 raft_consensus.cc:504] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.417292 3634 leader_election.cc:288] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3558df1427f94598a39b5bc6a0d2cdca (127.15.102.194:45575), 8ba3379278734825acdc38ad79f3c3a1 (127.15.102.193:34437)
I20260620 18:49:20.424396 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.417629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.194:45575 (local address 127.15.102.195:51629)
0620 18:49:20.417945 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.417968 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:20.418151 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:20.418615 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:20.418618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:20.418645 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:20.418943 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.418954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.420263 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.420269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.422473 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.422488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.423622 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.423629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.423795 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.423819 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 18:49:20.423928 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 18:49:20.424087 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260620 18:49:20.424377 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.417882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:51629 (local address 127.15.102.194:45575)
0620 18:49:20.418056 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.418061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.418085 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:49:20.418355 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.418360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.418458 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.418573 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:20.419146 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.420111 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.422637 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.423465 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.423640 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.423711 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.423955 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.424087 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:49:20.424842 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.417846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.102.195:60379 (local address 127.15.102.193:34437)
0620 18:49:20.418053 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.418060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.418732 (+ 672us) server_negotiation.cc:407] Connection header received
0620 18:49:20.418930 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.418935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.419036 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.419130 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 18:49:20.419832 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.420869 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.423162 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.423949 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.424150 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.424217 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.424485 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.424617 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:49:20.424894 3636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.417734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.15.102.195:60379)
0620 18:49:20.418601 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.418620 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:20.418759 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:20.419179 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:20.419184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:20.419213 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 18:49:20.419607 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.419619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.421082 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.421090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.422948 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:20.422958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.424150 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.424156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:20.424325 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.424346 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 18:49:20.424459 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 18:49:20.424594 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":528,"threads_started":1}
I20260620 18:49:20.425802 3433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a380afdf995b44f8908f9f5107f20636" candidate_uuid: "ff992e7c81474a15bf7d20b241382646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3558df1427f94598a39b5bc6a0d2cdca" is_pre_election: true
I20260620 18:49:20.425819 3302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a380afdf995b44f8908f9f5107f20636" candidate_uuid: "ff992e7c81474a15bf7d20b241382646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba3379278734825acdc38ad79f3c3a1" is_pre_election: true
I20260620 18:49:20.426314 3302 raft_consensus.cc:2454] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff992e7c81474a15bf7d20b241382646 in term 0.
I20260620 18:49:20.426311 3433 raft_consensus.cc:2454] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff992e7c81474a15bf7d20b241382646 in term 0.
I20260620 18:49:20.426986 3497 leader_election.cc:302] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3558df1427f94598a39b5bc6a0d2cdca, ff992e7c81474a15bf7d20b241382646; no voters:
I20260620 18:49:20.427430 3634 raft_consensus.cc:2781] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 18:49:20.427603 3634 raft_consensus.cc:482] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 18:49:20.427721 3634 raft_consensus.cc:3037] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:20.429867 3634 raft_consensus.cc:504] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.430456 3634 leader_election.cc:288] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [CANDIDATE]: Term 1 election: Requested vote from peers 3558df1427f94598a39b5bc6a0d2cdca (127.15.102.194:45575), 8ba3379278734825acdc38ad79f3c3a1 (127.15.102.193:34437)
I20260620 18:49:20.430956 3302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a380afdf995b44f8908f9f5107f20636" candidate_uuid: "ff992e7c81474a15bf7d20b241382646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba3379278734825acdc38ad79f3c3a1"
I20260620 18:49:20.430966 3433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a380afdf995b44f8908f9f5107f20636" candidate_uuid: "ff992e7c81474a15bf7d20b241382646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3558df1427f94598a39b5bc6a0d2cdca"
I20260620 18:49:20.431164 3302 raft_consensus.cc:3037] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:20.431177 3433 raft_consensus.cc:3037] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:20.433369 3302 raft_consensus.cc:2454] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff992e7c81474a15bf7d20b241382646 in term 1.
I20260620 18:49:20.433404 3433 raft_consensus.cc:2454] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff992e7c81474a15bf7d20b241382646 in term 1.
I20260620 18:49:20.433823 3498 leader_election.cc:302] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba3379278734825acdc38ad79f3c3a1, ff992e7c81474a15bf7d20b241382646; no voters:
I20260620 18:49:20.434160 3634 raft_consensus.cc:2781] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:20.434597 3634 raft_consensus.cc:686] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 1 LEADER]: Becoming Leader. State: Replica: ff992e7c81474a15bf7d20b241382646, State: Running, Role: LEADER
I20260620 18:49:20.435057 3634 consensus_queue.cc:237] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } }
I20260620 18:49:20.439765 3161 catalog_manager.cc:5495] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff992e7c81474a15bf7d20b241382646 (127.15.102.195). New cstate: current_term: 1 leader_uuid: "ff992e7c81474a15bf7d20b241382646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff992e7c81474a15bf7d20b241382646" member_type: VOTER last_known_addr { host: "127.15.102.195" port: 42327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 } health_report { overall_health: UNKNOWN } } }
I20260620 18:49:20.466055 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.460504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.15.102.195:42327)
0620 18:49:20.460706 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.460711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.460920 (+ 209us) server_negotiation.cc:407] Connection header received
0620 18:49:20.461083 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.461086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.461173 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.461298 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:20.461696 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.462567 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.464087 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.464485 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.464581 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.465006 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:20.465515 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:20.465639 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.465697 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.465760 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:49:20.474584 3302 raft_consensus.cc:1261] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 1 FOLLOWER]: Refusing update from remote peer ff992e7c81474a15bf7d20b241382646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:20.474584 3433 raft_consensus.cc:1261] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 1 FOLLOWER]: Refusing update from remote peer ff992e7c81474a15bf7d20b241382646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:20.475747 3637 consensus_queue.cc:1035] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba3379278734825acdc38ad79f3c3a1" member_type: VOTER last_known_addr { host: "127.15.102.193" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:20.476161 3634 consensus_queue.cc:1035] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3558df1427f94598a39b5bc6a0d2cdca" member_type: VOTER last_known_addr { host: "127.15.102.194" port: 45575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:49:20.486102 3610 tablet.cc:2369] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:20.488674 3640 mvcc.cc:205] Tried to move back new op lower bound from 7298995652493139968 to 7298995652346421248. Current Snapshot: MvccSnapshot[applied={T|T < 7298995652493139968}]
I20260620 18:49:20.504336 3160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:20.504693 3160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 18:49:20.508616 3160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 906bcf3816a642f594589df4fe905f2e: Sending DeleteTablet for 3 replicas of tablet a380afdf995b44f8908f9f5107f20636
I20260620 18:49:20.509819 3544 tablet_service.cc:1514] Processing DeleteTablet for tablet a380afdf995b44f8908f9f5107f20636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:20 UTC) from {username='slave'} at 127.0.0.1:50858
I20260620 18:49:20.509809 3413 tablet_service.cc:1514] Processing DeleteTablet for tablet a380afdf995b44f8908f9f5107f20636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:20 UTC) from {username='slave'} at 127.0.0.1:34610
I20260620 18:49:20.509938 3282 tablet_service.cc:1514] Processing DeleteTablet for tablet a380afdf995b44f8908f9f5107f20636 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 18:49:20 UTC) from {username='slave'} at 127.0.0.1:56582
I20260620 18:49:20.513763 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3220
I20260620 18:49:20.518884 3651 tablet_replica.cc:330] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1: stopping tablet replica
I20260620 18:49:20.519182 3649 tablet_replica.cc:330] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: stopping tablet replica
I20260620 18:49:20.519917 3649 raft_consensus.cc:2229] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 1 LEADER]: Raft consensus shutting down.
I20260620 18:49:20.520001 3651 raft_consensus.cc:2229] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:20.520632 3651 raft_consensus.cc:2258] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:20.520792 3649 raft_consensus.cc:2258] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:20.521119 3650 tablet_replica.cc:330] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: stopping tablet replica
I20260620 18:49:20.521521 3650 raft_consensus.cc:2229] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 18:49:20.522058 3650 raft_consensus.cc:2258] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 18:49:20.523167 3649 ts_tablet_manager.cc:1905] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:20.524250 3650 ts_tablet_manager.cc:1905] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:20.529016 3651 ts_tablet_manager.cc:1905] T a380afdf995b44f8908f9f5107f20636 P 8ba3379278734825acdc38ad79f3c3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 18:49:20.529078 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.515522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.15.102.193:34437)
0620 18:49:20.515785 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.515795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.520844 (+ 5049us) server_negotiation.cc:407] Connection header received
0620 18:49:20.521044 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.521050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.521153 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.521266 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:20.521893 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.523113 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.525025 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.525686 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.525790 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.527308 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:20.527324 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:20.527344 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:20.528146 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:20.528207 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:20.528248 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:20.528334 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.528591 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.528705 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 18:49:20.530076 3342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:20.530795 3649 ts_tablet_manager.cc:1918] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:20.531031 3649 log.cc:1196] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-2/wal/wals/a380afdf995b44f8908f9f5107f20636
I20260620 18:49:20.531733 3649 ts_tablet_manager.cc:1939] T a380afdf995b44f8908f9f5107f20636 P ff992e7c81474a15bf7d20b241382646: Deleting consensus metadata
I20260620 18:49:20.533644 3147 catalog_manager.cc:4841] TS ff992e7c81474a15bf7d20b241382646 (127.15.102.195:42327): tablet a380afdf995b44f8908f9f5107f20636 (table test-table [id=c3c9c224398f45899b264338ef4ebe4e]) successfully deleted
I20260620 18:49:20.534222 3650 ts_tablet_manager.cc:1918] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 18:49:20.534415 3650 log.cc:1196] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: Deleting WAL directory at /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781981043007005-15771-0/minicluster-data/ts-1/wal/wals/a380afdf995b44f8908f9f5107f20636
I20260620 18:49:20.534875 3650 ts_tablet_manager.cc:1939] T a380afdf995b44f8908f9f5107f20636 P 3558df1427f94598a39b5bc6a0d2cdca: Deleting consensus metadata
I20260620 18:49:20.536512 3145 catalog_manager.cc:4841] TS 3558df1427f94598a39b5bc6a0d2cdca (127.15.102.194:45575): tablet a380afdf995b44f8908f9f5107f20636 (table test-table [id=c3c9c224398f45899b264338ef4ebe4e]) successfully deleted
I20260620 18:49:20.628067 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 3220
W20260620 18:49:20.652011 3146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.102.193:34437 (error 108)
I20260620 18:49:20.652642 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3351
I20260620 18:49:20.654304 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.653792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:41652)
0620 18:49:20.653986 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.654116 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 18:49:20.654553 3146 catalog_manager.cc:4584] TS 8ba3379278734825acdc38ad79f3c3a1 (127.15.102.193:34437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a380afdf995b44f8908f9f5107f20636: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
I20260620 18:49:20.664805 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.653578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.15.102.194:45575)
0620 18:49:20.653783 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.653790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.654018 (+ 228us) server_negotiation.cc:407] Connection header received
0620 18:49:20.654229 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.654236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.654343 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.654506 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:20.655137 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.656107 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.657661 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.658148 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.658333 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.663321 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:20.663335 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:20.663358 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:20.663979 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:20.664033 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:20.664056 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:20.664218 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.664348 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.664472 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:49:20.665536 3474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:20.708258 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.707774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:41662)
0620 18:49:20.707991 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.708079 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 18:49:20.709422 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.709058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:41664)
0620 18:49:20.709215 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.709292 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:49:20.770131 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.769542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:41674)
0620 18:49:20.769799 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.769914 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 18:49:20.771468 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.770973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:41690)
0620 18:49:20.771152 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.771291 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:49:20.794873 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 3351
I20260620 18:49:20.812114 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3482
I20260620 18:49:20.819657 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.813198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.15.102.195:42327)
0620 18:49:20.813449 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:20.813454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:20.813473 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:49:20.813529 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:20.813532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:20.813605 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:20.813735 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:20.814158 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.815004 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.816379 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:20.816835 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:20.816994 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:20.818142 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:20.818151 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:20.818167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:20.818787 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:20.818835 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:20.818854 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:20.819002 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:20.819131 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 18:49:20.819229 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 18:49:20.820452 3604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:20.852370 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.851788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:41694)
0620 18:49:20.852034 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.852161 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:49:20.853487 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:20.853049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:41700)
0620 18:49:20.853268 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:20.853354 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 18:49:20.941154 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 3482
I20260620 18:49:20.960414 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3130
I20260620 18:49:20.961531 3191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:21.071848 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:21.071250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.193:34437 (local address 127.0.0.1:41702)
0620 18:49:21.071465 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:21.071606 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.193:34437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 18:49:21.072284 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 3130
2026-06-20T18:49:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2871 ms)
[----------] 6 tests from EncryptionPolicyTest (22417 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:21 dist-test-slave-zfp8 krb5kdc[3660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:21 dist-test-slave-zfp8 krb5kdc[3660](info): set up 2 sockets
Jun 20 18:49:21 dist-test-slave-zfp8 krb5kdc[3660](info): commencing operation
krb5kdc: starting...
W20260620 18:49:22.287645 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:22 dist-test-slave-zfp8 krb5kdc[3660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:22.554692 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:22.884644 3673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:22.884932 3673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:22.885097 3673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:22.892463 3673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:22.892581 3673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:22.892688 3673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:22.892793 3673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:22.892879 3673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:22.902925 3673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:22.904282 3673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:22.905655 3673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:22.917856 3678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:22.917994 3679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:22.917966 3681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:22.919130 3673 server_base.cc:1034] running on GCE node
I20260620 18:49:22.920046 3673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:22.921564 3673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:22.922825 3673 hybrid_clock.cc:648] HybridClock initialized: now 1781981362922779 us; error 43 us; skew 500 ppm
I20260620 18:49:22.923202 3673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:22 dist-test-slave-zfp8 krb5kdc[3660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:22.926587 3673 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:22.928287 3673 webserver.cc:460] Webserver started at http://127.0.0.1:41521/ using document root <none> and password file <none>
I20260620 18:49:22.929014 3673 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:22.929175 3673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:22.929492 3673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:22.932828 3673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "c930be7a33384b179929767c5c441227"
format_stamp: "Formatted at 2026-06-20 18:49:22 on dist-test-slave-zfp8"
server_key: "37d5dd63ea742936b14c5c3e207b1428"
server_key_iv: "441e999d3a6aff53f981dede279e9341"
server_key_version: "encryptionkey@0"
I20260620 18:49:22.933656 3673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "c930be7a33384b179929767c5c441227"
format_stamp: "Formatted at 2026-06-20 18:49:22 on dist-test-slave-zfp8"
server_key: "37d5dd63ea742936b14c5c3e207b1428"
server_key_iv: "441e999d3a6aff53f981dede279e9341"
server_key_version: "encryptionkey@0"
I20260620 18:49:22.939069 3673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:49:22.942654 3688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:22.944104 3673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:22.944299 3673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "c930be7a33384b179929767c5c441227"
format_stamp: "Formatted at 2026-06-20 18:49:22 on dist-test-slave-zfp8"
server_key: "37d5dd63ea742936b14c5c3e207b1428"
server_key_iv: "441e999d3a6aff53f981dede279e9341"
server_key_version: "encryptionkey@0"
I20260620 18:49:22.944504 3673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:22.965685 3673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:22.969913 3673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:22.970228 3673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:22.989601 3673 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35879
I20260620 18:49:22.989701 3739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35879 every 8 connection(s)
I20260620 18:49:22.991012 3673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:22.994920 3740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:23.000078 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 3673
I20260620 18:49:23.000345 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:23.000982 15771 external_mini_cluster.cc:1428] Setting key 1dfff749c05e031c9b6676140a513e02
I20260620 18:49:23.006734 3740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227: Bootstrap starting.
I20260620 18:49:23.010237 3740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:23.011478 3740 log.cc:824] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:23.015113 3740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227: No bootstrap required, opened a new log
Jun 20 18:49:23 dist-test-slave-zfp8 krb5kdc[3660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:49:23.026098 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:23.003055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54435 (local address 127.0.0.1:35879)
0620 18:49:23.003929 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:23.003941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:23.003983 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:49:23.005716 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:23.005740 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:23.006140 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:23.006544 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:23.008132 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:23.009498 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:23.010912 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:23.011762 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:23.011958 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:23.016449 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:23.016462 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:23.016481 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:23.016535 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:23.020899 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:23.021710 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:23.021719 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:23.021729 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:23.021963 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:23.022415 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:23.022421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:23.022425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:23.023193 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:23.023568 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:23.024106 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 18:49:23.024501 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":401,"threads_started":1}
I20260620 18:49:23.027004 3740 raft_consensus.cc:348] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c930be7a33384b179929767c5c441227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } }
I20260620 18:49:23.027534 3740 raft_consensus.cc:374] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:23.027658 3740 raft_consensus.cc:729] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c930be7a33384b179929767c5c441227, State: Initialized, Role: FOLLOWER
I20260620 18:49:23.028354 3740 consensus_queue.cc:260] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c930be7a33384b179929767c5c441227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } }
I20260620 18:49:23.028641 3740 raft_consensus.cc:388] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:23.028803 3740 raft_consensus.cc:482] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:23.029016 3740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:23.030463 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:23.031826 3740 raft_consensus.cc:504] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c930be7a33384b179929767c5c441227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } }
I20260620 18:49:23.032377 3740 leader_election.cc:302] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c930be7a33384b179929767c5c441227; no voters:
I20260620 18:49:23.033044 3740 leader_election.cc:288] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:23.033186 3745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:23.033953 3745 raft_consensus.cc:686] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [term 1 LEADER]: Becoming Leader. State: Replica: c930be7a33384b179929767c5c441227, State: Running, Role: LEADER
I20260620 18:49:23.034749 3745 consensus_queue.cc:237] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c930be7a33384b179929767c5c441227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } }
I20260620 18:49:23.037092 3745 sys_catalog.cc:455] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c930be7a33384b179929767c5c441227. Latest consensus state: current_term: 1 leader_uuid: "c930be7a33384b179929767c5c441227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c930be7a33384b179929767c5c441227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } } }
I20260620 18:49:23.038986 3745 sys_catalog.cc:458] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:23.038547 3749 sys_catalog.cc:455] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c930be7a33384b179929767c5c441227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c930be7a33384b179929767c5c441227" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } } }
I20260620 18:49:23.039469 3749 sys_catalog.cc:458] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:23.045387 3740 sys_catalog.cc:564] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:23.045573 3751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:23.053283 3751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:23.064747 3743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:23.044402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34353 (local address 127.0.0.1:35879)
0620 18:49:23.044565 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:23.044570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:23.045133 (+ 563us) server_negotiation.cc:407] Connection header received
0620 18:49:23.045344 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:23.045350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:23.045428 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:23.045545 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:23.047157 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:23.048540 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:23.050234 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:23.050679 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:23.050835 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:23.054579 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:23.054590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:23.054597 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:23.054652 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:23.059742 (+ 5090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:23.061113 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:23.061122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:23.061128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:23.061324 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:23.062111 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:23.062119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:23.062125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:23.062644 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:23.062968 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:23.063913 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 18:49:23.064172 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 18:49:23.074688 3751 catalog_manager.cc:1347] Generated new cluster ID: 894b7710edd64fd799037ebd683769ef
I20260620 18:49:23.074829 3751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:23.092869 3751 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:23.094195 3751 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:23.104877 3751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c930be7a33384b179929767c5c441227: Generated new TSK 0
I20260620 18:49:23.105729 3751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:23.118650 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3673
I20260620 18:49:23.120373 3735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:23.212219 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 3673
W20260620 18:49:23.229907 3658 connection.cc:535] client connection to 127.0.0.1:35879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35879): Connection reset by peer (error 104)
2026-06-20T18:49:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2134 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:23 dist-test-slave-zfp8 krb5kdc[3772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:23 dist-test-slave-zfp8 krb5kdc[3772](info): set up 2 sockets
Jun 20 18:49:23 dist-test-slave-zfp8 krb5kdc[3772](info): commencing operation
krb5kdc: starting...
W20260620 18:49:24.331669 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:24 dist-test-slave-zfp8 krb5kdc[3772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:24.588013 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:24.926003 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:24.926308 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:24.926401 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:24.933684 3785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:24.933800 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:24.933846 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:24.933900 3785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:24.933946 3785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:24.944034 3785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:24.945338 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:24.946646 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:24.959090 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:24.959190 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:24.959216 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:24.959839 3785 server_base.cc:1034] running on GCE node
I20260620 18:49:24.960562 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:24.961958 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:24.963162 3785 hybrid_clock.cc:648] HybridClock initialized: now 1781981364963131 us; error 27 us; skew 500 ppm
I20260620 18:49:24.963580 3785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:24 dist-test-slave-zfp8 krb5kdc[3772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:24.966648 3785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:24.968223 3785 webserver.cc:460] Webserver started at http://127.0.0.1:34143/ using document root <none> and password file <none>
I20260620 18:49:24.968852 3785 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:24.968950 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:24.969202 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:24.972230 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "b962f716a9e448b980cafe67a59d2f7f"
format_stamp: "Formatted at 2026-06-20 18:49:24 on dist-test-slave-zfp8"
server_key: "5aafc0752efacf85b680738caddcc4c0"
server_key_iv: "7892cad663383c831424ad466400fec5"
server_key_version: "encryptionkey@0"
I20260620 18:49:24.972946 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "b962f716a9e448b980cafe67a59d2f7f"
format_stamp: "Formatted at 2026-06-20 18:49:24 on dist-test-slave-zfp8"
server_key: "5aafc0752efacf85b680738caddcc4c0"
server_key_iv: "7892cad663383c831424ad466400fec5"
server_key_version: "encryptionkey@0"
I20260620 18:49:24.978245 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 18:49:24.981678 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:24.983011 3785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 18:49:24.983196 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "b962f716a9e448b980cafe67a59d2f7f"
format_stamp: "Formatted at 2026-06-20 18:49:24 on dist-test-slave-zfp8"
server_key: "5aafc0752efacf85b680738caddcc4c0"
server_key_iv: "7892cad663383c831424ad466400fec5"
server_key_version: "encryptionkey@0"
I20260620 18:49:24.983464 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:24.998848 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:25.002950 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:25.003319 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:25.022053 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46055
I20260620 18:49:25.022075 3851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46055 every 8 connection(s)
I20260620 18:49:25.023531 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:25.024762 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 3785
I20260620 18:49:25.024945 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:25.025508 15771 external_mini_cluster.cc:1428] Setting key 7085ea5f04d0e5af9caa59a687f6eeea
I20260620 18:49:25.028134 3852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:25.039011 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:25.027361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41527 (local address 127.0.0.1:46055)
0620 18:49:25.028163 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:25.028175 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:25.028253 (+ 78us) server_negotiation.cc:407] Connection header received
0620 18:49:25.029893 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:25.029913 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:25.030307 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:25.030649 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:25.031227 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:25.032633 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:25.034485 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:25.035124 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:25.035339 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:25.036626 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:25.036634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:25.036648 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:25.037224 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:25.037301 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:25.037328 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:25.037457 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:25.037813 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 18:49:25.038002 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":379,"threads_started":1}
I20260620 18:49:25.040275 3852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f: Bootstrap starting.
I20260620 18:49:25.042953 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:25.043597 3852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:25.044806 3852 log.cc:824] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:25.048344 3852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f: No bootstrap required, opened a new log
I20260620 18:49:25.055042 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:25.048103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53579 (local address 127.0.0.1:46055)
0620 18:49:25.048291 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:25.048299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:25.048684 (+ 385us) server_negotiation.cc:407] Connection header received
0620 18:49:25.048831 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:25.048838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:25.048931 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:25.049060 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:25.049635 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:25.050825 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:25.052287 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:25.052962 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:25.053085 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:25.054216 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:25.054223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:25.054228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:25.054251 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:25.054305 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:25.054325 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:25.054435 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:25.054649 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 18:49:25.054714 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:49:25.057848 3852 raft_consensus.cc:348] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b962f716a9e448b980cafe67a59d2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46055 } }
I20260620 18:49:25.058260 3852 raft_consensus.cc:374] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:25.058393 3852 raft_consensus.cc:729] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b962f716a9e448b980cafe67a59d2f7f, State: Initialized, Role: FOLLOWER
I20260620 18:49:25.058984 3852 consensus_queue.cc:260] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b962f716a9e448b980cafe67a59d2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46055 } }
I20260620 18:49:25.059193 3852 raft_consensus.cc:388] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:25.059429 3852 raft_consensus.cc:482] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:25.059615 3852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:25.062247 3852 raft_consensus.cc:504] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b962f716a9e448b980cafe67a59d2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46055 } }
I20260620 18:49:25.062780 3852 leader_election.cc:302] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b962f716a9e448b980cafe67a59d2f7f; no voters:
I20260620 18:49:25.063432 3852 leader_election.cc:288] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:25.063586 3862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:25.064294 3862 raft_consensus.cc:686] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [term 1 LEADER]: Becoming Leader. State: Replica: b962f716a9e448b980cafe67a59d2f7f, State: Running, Role: LEADER
I20260620 18:49:25.064903 3862 consensus_queue.cc:237] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b962f716a9e448b980cafe67a59d2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46055 } }
I20260620 18:49:25.065671 3852 sys_catalog.cc:564] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:25.067945 3864 sys_catalog.cc:455] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b962f716a9e448b980cafe67a59d2f7f. Latest consensus state: current_term: 1 leader_uuid: "b962f716a9e448b980cafe67a59d2f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b962f716a9e448b980cafe67a59d2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46055 } } }
I20260620 18:49:25.068420 3864 sys_catalog.cc:458] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:25.068981 3863 sys_catalog.cc:455] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b962f716a9e448b980cafe67a59d2f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b962f716a9e448b980cafe67a59d2f7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46055 } } }
I20260620 18:49:25.069347 3863 sys_catalog.cc:458] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:25.070621 3870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:25.076388 3870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:25.086043 3870 catalog_manager.cc:1347] Generated new cluster ID: cc9dddc47fdc4370bff2c585761051af
I20260620 18:49:25.086177 3870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:25.108261 3870 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:25.109701 3870 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:25.115899 3870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b962f716a9e448b980cafe67a59d2f7f: Generated new TSK 0
I20260620 18:49:25.117017 3870 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:25.133476 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3785
W20260620 18:49:25.134865 3847 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41527
W20260620 18:49:25.135795 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3785: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:25.136016 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 3785
W20260620 18:49:25.154537 3770 connection.cc:535] client connection to 127.0.0.1:46055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46055): Connection reset by peer (error 104)
2026-06-20T18:49:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:25 dist-test-slave-zfp8 krb5kdc[3883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:25 dist-test-slave-zfp8 krb5kdc[3883](info): set up 2 sockets
Jun 20 18:49:25 dist-test-slave-zfp8 krb5kdc[3883](info): commencing operation
krb5kdc: starting...
W20260620 18:49:26.239890 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:26 dist-test-slave-zfp8 krb5kdc[3883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:26.506560 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:26.833758 3896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:26.834051 3896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:26.834246 3896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:26.841418 3896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:26.841528 3896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:26.841575 3896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:26.841681 3896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:26.841780 3896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:26.851755 3896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:26.853113 3896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:26.854466 3896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:26.866573 3904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:26.866539 3901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:26.866526 3902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:26.866966 3896 server_base.cc:1034] running on GCE node
I20260620 18:49:26.868085 3896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:26.869534 3896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:26.870797 3896 hybrid_clock.cc:648] HybridClock initialized: now 1781981366870754 us; error 59 us; skew 500 ppm
I20260620 18:49:26.871176 3896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:26 dist-test-slave-zfp8 krb5kdc[3883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:26.879366 3896 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:26.881032 3896 webserver.cc:460] Webserver started at http://127.0.0.1:38211/ using document root <none> and password file <none>
I20260620 18:49:26.881765 3896 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:26.881978 3896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:26.882319 3896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:26.885397 3896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "4180f77b96dc4dfe9621912a626a8ff7"
format_stamp: "Formatted at 2026-06-20 18:49:26 on dist-test-slave-zfp8"
server_key: "c99dc2dc23d0e3610a75bd52e51ffc34"
server_key_iv: "baf526eb46783bad2b86ff0c4d385bee"
server_key_version: "encryptionkey@0"
I20260620 18:49:26.886133 3896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "4180f77b96dc4dfe9621912a626a8ff7"
format_stamp: "Formatted at 2026-06-20 18:49:26 on dist-test-slave-zfp8"
server_key: "c99dc2dc23d0e3610a75bd52e51ffc34"
server_key_iv: "baf526eb46783bad2b86ff0c4d385bee"
server_key_version: "encryptionkey@0"
I20260620 18:49:26.891475 3896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 18:49:26.895033 3911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:26.896588 3896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:26.896733 3896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "4180f77b96dc4dfe9621912a626a8ff7"
format_stamp: "Formatted at 2026-06-20 18:49:26 on dist-test-slave-zfp8"
server_key: "c99dc2dc23d0e3610a75bd52e51ffc34"
server_key_iv: "baf526eb46783bad2b86ff0c4d385bee"
server_key_version: "encryptionkey@0"
I20260620 18:49:26.896903 3896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:26.902555 3896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:26.906442 3896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:26.906692 3896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:26.925531 3896 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44079
I20260620 18:49:26.925635 3962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44079 every 8 connection(s)
I20260620 18:49:26.927011 3896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:26.930896 3963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:26.931821 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 3896
I20260620 18:49:26.932029 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:26.932492 15771 external_mini_cluster.cc:1428] Setting key e3b7e8f609fac94b205f9778cf35d61e
I20260620 18:49:26.942219 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:26.934453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49907 (local address 127.0.0.1:44079)
0620 18:49:26.935322 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:26.935341 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:26.935387 (+ 46us) server_negotiation.cc:407] Connection header received
0620 18:49:26.938207 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:26.938235 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:26.938729 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:26.939203 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:26.940239 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:26.940249 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:26.940266 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:26.940283 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:26.940365 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:26.940400 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:26.940640 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:26.940773 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 18:49:26.940966 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":353,"threads_started":1}
I20260620 18:49:26.946105 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:26.946678 3963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7: Bootstrap starting.
I20260620 18:49:26.949977 3963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:26.951509 3963 log.cc:824] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:26.953310 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:26.950547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40327 (local address 127.0.0.1:44079)
0620 18:49:26.950700 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:26.950704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:26.951171 (+ 467us) server_negotiation.cc:407] Connection header received
0620 18:49:26.951324 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:26.951338 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:26.951400 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:26.951540 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:26.952576 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:26.952580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:26.952583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:26.952598 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:26.952639 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:26.952655 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:26.952750 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:26.952921 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:49:26.952973 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:49:26.954769 3963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7: No bootstrap required, opened a new log
I20260620 18:49:26.961962 3963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4180f77b96dc4dfe9621912a626a8ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44079 } }
I20260620 18:49:26.962391 3963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:26.962508 3963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4180f77b96dc4dfe9621912a626a8ff7, State: Initialized, Role: FOLLOWER
I20260620 18:49:26.963093 3963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4180f77b96dc4dfe9621912a626a8ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44079 } }
I20260620 18:49:26.963392 3963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:26.963570 3963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:26.963760 3963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:26.966100 3963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4180f77b96dc4dfe9621912a626a8ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44079 } }
I20260620 18:49:26.966637 3963 leader_election.cc:302] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4180f77b96dc4dfe9621912a626a8ff7; no voters:
I20260620 18:49:26.967271 3963 leader_election.cc:288] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:26.967437 3973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:26.968112 3973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 4180f77b96dc4dfe9621912a626a8ff7, State: Running, Role: LEADER
I20260620 18:49:26.968698 3973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4180f77b96dc4dfe9621912a626a8ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44079 } }
I20260620 18:49:26.969558 3963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:26.971553 3974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4180f77b96dc4dfe9621912a626a8ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4180f77b96dc4dfe9621912a626a8ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44079 } } }
I20260620 18:49:26.972033 3974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:26.971701 3975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4180f77b96dc4dfe9621912a626a8ff7. Latest consensus state: current_term: 1 leader_uuid: "4180f77b96dc4dfe9621912a626a8ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4180f77b96dc4dfe9621912a626a8ff7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44079 } } }
I20260620 18:49:26.972808 3975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:26.974064 3980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:26.979327 3980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:26.989296 3980 catalog_manager.cc:1347] Generated new cluster ID: 7e47b8d2a20b47c0bdb1969b1c732196
I20260620 18:49:26.989414 3980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:27.002951 3980 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:27.003479 3980 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:27.013146 3980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4180f77b96dc4dfe9621912a626a8ff7: Generated new TSK 0
I20260620 18:49:27.014173 3980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:27.025285 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3896
W20260620 18:49:27.026518 3958 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49907
W20260620 18:49:27.027284 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 3896: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:27.027433 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 3896
2026-06-20T18:49:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1889 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:27 dist-test-slave-zfp8 krb5kdc[3994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:27 dist-test-slave-zfp8 krb5kdc[3994](info): set up 2 sockets
Jun 20 18:49:27 dist-test-slave-zfp8 krb5kdc[3994](info): commencing operation
krb5kdc: starting...
W20260620 18:49:28.135897 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:28 dist-test-slave-zfp8 krb5kdc[3994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:28.394588 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:28.734684 4007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:28.735059 4007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:28.735311 4007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:28.742770 4007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:28.742949 4007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:28.743072 4007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:28.743199 4007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:28.743321 4007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:28.753460 4007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:28.754882 4007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:28.756385 4007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:28.768675 4015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:28.768772 4013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:28.768749 4012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:28.769845 4007 server_base.cc:1034] running on GCE node
I20260620 18:49:28.770640 4007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:28.772137 4007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:28.773350 4007 hybrid_clock.cc:648] HybridClock initialized: now 1781981368773309 us; error 34 us; skew 500 ppm
I20260620 18:49:28.773675 4007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:28 dist-test-slave-zfp8 krb5kdc[3994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:28.776811 4007 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:28.778394 4007 webserver.cc:460] Webserver started at http://127.0.0.1:37885/ using document root <none> and password file <none>
I20260620 18:49:28.779112 4007 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:28.779273 4007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:28.779619 4007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:28.782658 4007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "6886952fdbb84561ae0b10afe1fb3a1b"
format_stamp: "Formatted at 2026-06-20 18:49:28 on dist-test-slave-zfp8"
server_key: "acf043e11325aae07c032915dbe57770"
server_key_iv: "c6297275f02c32203adc9230d5a52505"
server_key_version: "encryptionkey@0"
I20260620 18:49:28.783545 4007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "6886952fdbb84561ae0b10afe1fb3a1b"
format_stamp: "Formatted at 2026-06-20 18:49:28 on dist-test-slave-zfp8"
server_key: "acf043e11325aae07c032915dbe57770"
server_key_iv: "c6297275f02c32203adc9230d5a52505"
server_key_version: "encryptionkey@0"
I20260620 18:49:28.788877 4007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:49:28.792363 4022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:28.793756 4007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:28.793978 4007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "6886952fdbb84561ae0b10afe1fb3a1b"
format_stamp: "Formatted at 2026-06-20 18:49:28 on dist-test-slave-zfp8"
server_key: "acf043e11325aae07c032915dbe57770"
server_key_iv: "c6297275f02c32203adc9230d5a52505"
server_key_version: "encryptionkey@0"
I20260620 18:49:28.794204 4007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:28.816147 4007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:28.820375 4007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:28.820690 4007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:28.839986 4007 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44807
I20260620 18:49:28.840021 4073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44807 every 8 connection(s)
I20260620 18:49:28.841423 4007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:28.841807 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4007
I20260620 18:49:28.841996 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:28.842535 15771 external_mini_cluster.cc:1428] Setting key 86da69cb390f80ca5629033ff1cf5d5a
I20260620 18:49:28.845911 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:49:28 dist-test-slave-zfp8 krb5kdc[3994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:49:28.859190 4074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b: Bootstrap starting.
I20260620 18:49:28.862540 4074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:28.864038 4074 log.cc:824] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:28.866012 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:28.844424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50697 (local address 127.0.0.1:44807)
0620 18:49:28.845207 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:28.845218 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:28.845260 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:49:28.846958 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:28.846981 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:28.847431 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:28.847748 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:28.849564 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:28.850898 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:28.852803 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:28.853475 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:28.853610 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:28.858586 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:28.858594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:28.858609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:28.858652 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:28.861765 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:28.862564 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:28.862571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:28.862577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:28.862804 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:28.863251 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:28.863259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:28.863263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:28.863842 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:28.864154 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:28.864611 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 18:49:28.865044 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":375,"threads_started":1}
I20260620 18:49:28.867118 4074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b: No bootstrap required, opened a new log
I20260620 18:49:28.870095 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:28.875401 4074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6886952fdbb84561ae0b10afe1fb3a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260620 18:49:28.875947 4074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:28.876082 4074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6886952fdbb84561ae0b10afe1fb3a1b, State: Initialized, Role: FOLLOWER
I20260620 18:49:28.876750 4074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6886952fdbb84561ae0b10afe1fb3a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260620 18:49:28.877053 4074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:28.877192 4074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:28.877396 4074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:28.879776 4074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6886952fdbb84561ae0b10afe1fb3a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260620 18:49:28.880334 4074 leader_election.cc:302] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6886952fdbb84561ae0b10afe1fb3a1b; no voters:
I20260620 18:49:28.881026 4074 leader_election.cc:288] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:28.881117 4084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:28.883291 4084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [term 1 LEADER]: Becoming Leader. State: Replica: 6886952fdbb84561ae0b10afe1fb3a1b, State: Running, Role: LEADER
I20260620 18:49:28.883664 4074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:28.883948 4084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6886952fdbb84561ae0b10afe1fb3a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260620 18:49:28.888734 4085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6886952fdbb84561ae0b10afe1fb3a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6886952fdbb84561ae0b10afe1fb3a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260620 18:49:28.889205 4085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:28.889813 4086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6886952fdbb84561ae0b10afe1fb3a1b. Latest consensus state: current_term: 1 leader_uuid: "6886952fdbb84561ae0b10afe1fb3a1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6886952fdbb84561ae0b10afe1fb3a1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260620 18:49:28.890141 4086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:28.891563 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:28.874541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51955 (local address 127.0.0.1:44807)
0620 18:49:28.874734 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:28.874740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:28.875364 (+ 624us) server_negotiation.cc:407] Connection header received
0620 18:49:28.875522 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:28.875528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:28.875607 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:28.875719 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:28.877568 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:28.878339 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:28.879694 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:28.880221 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:28.880401 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:28.883088 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:28.883097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:28.883124 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:28.883171 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:28.887058 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:28.887987 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:28.887992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:28.887995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:28.888232 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:28.888701 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:28.888706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:28.888709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:28.889106 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:28.889404 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:28.890707 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0620 18:49:28.891026 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:49:28.896435 4094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:28.902014 4094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:28.911065 4094 catalog_manager.cc:1347] Generated new cluster ID: 09bf8a26638444faa75ceac6035ab0be
I20260620 18:49:28.911191 4094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:28.945080 4094 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:28.946444 4094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:28.958151 4094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6886952fdbb84561ae0b10afe1fb3a1b: Generated new TSK 0
I20260620 18:49:28.959080 4094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:28.966665 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4007
I20260620 18:49:28.968164 4069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:29.063129 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4007
2026-06-20T18:49:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:29 dist-test-slave-zfp8 krb5kdc[4107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:29 dist-test-slave-zfp8 krb5kdc[4107](info): set up 2 sockets
Jun 20 18:49:29 dist-test-slave-zfp8 krb5kdc[4107](info): commencing operation
krb5kdc: starting...
W20260620 18:49:30.167692 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:30 dist-test-slave-zfp8 krb5kdc[4107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:30.422741 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:30.753996 4120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:30.754388 4120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:30.754557 4120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:30.762041 4120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:30.762240 4120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:30.762351 4120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:30.762462 4120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:30.762542 4120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:30.773015 4120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:30.774451 4120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:30.775879 4120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:30.788443 4128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:30.788590 4125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:30.788511 4126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:30.789865 4120 server_base.cc:1034] running on GCE node
I20260620 18:49:30.790745 4120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:30.792452 4120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:30.793706 4120 hybrid_clock.cc:648] HybridClock initialized: now 1781981370793662 us; error 48 us; skew 500 ppm
I20260620 18:49:30.794108 4120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:30 dist-test-slave-zfp8 krb5kdc[4107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:30.797515 4120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:30.799105 4120 webserver.cc:460] Webserver started at http://127.0.0.1:39467/ using document root <none> and password file <none>
I20260620 18:49:30.799871 4120 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:30.800014 4120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:30.800328 4120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:30.803411 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "eafceac0893c46c1a400a8561414bf65"
format_stamp: "Formatted at 2026-06-20 18:49:30 on dist-test-slave-zfp8"
server_key: "d69e916c34f2dad3e100243b85988dff"
server_key_iv: "6727fac565177ea346a6940ff885a651"
server_key_version: "encryptionkey@0"
I20260620 18:49:30.804118 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "eafceac0893c46c1a400a8561414bf65"
format_stamp: "Formatted at 2026-06-20 18:49:30 on dist-test-slave-zfp8"
server_key: "d69e916c34f2dad3e100243b85988dff"
server_key_iv: "6727fac565177ea346a6940ff885a651"
server_key_version: "encryptionkey@0"
I20260620 18:49:30.809180 4120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:49:30.812621 4135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:30.814042 4120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:30.814261 4120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "eafceac0893c46c1a400a8561414bf65"
format_stamp: "Formatted at 2026-06-20 18:49:30 on dist-test-slave-zfp8"
server_key: "d69e916c34f2dad3e100243b85988dff"
server_key_iv: "6727fac565177ea346a6940ff885a651"
server_key_version: "encryptionkey@0"
I20260620 18:49:30.814481 4120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:30.843678 4120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:30.847999 4120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:30.848333 4120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:30.867838 4120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46795
I20260620 18:49:30.867949 4186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46795 every 8 connection(s)
I20260620 18:49:30.869306 4120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:30.873070 4187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:30.878331 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4120
I20260620 18:49:30.878515 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:30.879025 15771 external_mini_cluster.cc:1428] Setting key fcb4bb461ed8f0f9cb2a0e11afb2a7d5
I20260620 18:49:30.884249 4187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65: Bootstrap starting.
I20260620 18:49:30.887880 4187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:30.889024 4187 log.cc:824] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:30.892266 4187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65: No bootstrap required, opened a new log
I20260620 18:49:30.893254 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:30.880966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50197 (local address 127.0.0.1:46795)
0620 18:49:30.881960 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:30.881973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:30.882024 (+ 51us) server_negotiation.cc:407] Connection header received
0620 18:49:30.883886 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:30.883915 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:30.884310 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:30.884708 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:30.885287 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:30.886624 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:30.888495 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:30.889251 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:30.889409 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:30.890723 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:30.890738 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:30.890752 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:30.891349 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:30.891434 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:30.891460 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:30.891555 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:30.891796 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 18:49:30.892071 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":461,"threads_started":1}
I20260620 18:49:30.897109 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:30.900619 4187 raft_consensus.cc:348] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafceac0893c46c1a400a8561414bf65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260620 18:49:30.900998 4187 raft_consensus.cc:374] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:30.901120 4187 raft_consensus.cc:729] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafceac0893c46c1a400a8561414bf65, State: Initialized, Role: FOLLOWER
I20260620 18:49:30.901859 4187 consensus_queue.cc:260] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafceac0893c46c1a400a8561414bf65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260620 18:49:30.902191 4187 raft_consensus.cc:388] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:30.902350 4187 raft_consensus.cc:482] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:30.902560 4187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:30.905356 4187 raft_consensus.cc:504] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafceac0893c46c1a400a8561414bf65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260620 18:49:30.905869 4187 leader_election.cc:302] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eafceac0893c46c1a400a8561414bf65; no voters:
I20260620 18:49:30.906551 4187 leader_election.cc:288] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:30.906656 4197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:30.908552 4197 raft_consensus.cc:686] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [term 1 LEADER]: Becoming Leader. State: Replica: eafceac0893c46c1a400a8561414bf65, State: Running, Role: LEADER
I20260620 18:49:30.908680 4190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:30.901616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33863 (local address 127.0.0.1:46795)
0620 18:49:30.901801 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:30.901808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:30.902182 (+ 374us) server_negotiation.cc:407] Connection header received
0620 18:49:30.902409 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:30.902414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:30.902512 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:30.902627 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:30.903160 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:30.904287 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:30.905631 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:30.906221 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:30.906410 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:30.907743 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:30.907755 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:30.907760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:30.907779 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:30.907819 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:30.907836 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:30.907889 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:30.908097 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:49:30.908190 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 18:49:30.909096 4187 sys_catalog.cc:564] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:30.909381 4197 consensus_queue.cc:237] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafceac0893c46c1a400a8561414bf65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260620 18:49:30.912634 4198 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eafceac0893c46c1a400a8561414bf65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafceac0893c46c1a400a8561414bf65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } } }
I20260620 18:49:30.913080 4198 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:30.913661 4199 sys_catalog.cc:455] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eafceac0893c46c1a400a8561414bf65. Latest consensus state: current_term: 1 leader_uuid: "eafceac0893c46c1a400a8561414bf65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eafceac0893c46c1a400a8561414bf65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } } }
I20260620 18:49:30.913944 4199 sys_catalog.cc:458] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:30.915987 4209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:30.921099 4209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:30.922075 4214 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:49:30.929778 4209 catalog_manager.cc:1347] Generated new cluster ID: fa3b1190af04411d868d6468abcb05a2
I20260620 18:49:30.929903 4209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:30.941407 4209 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:30.942721 4209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:30.952013 4209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eafceac0893c46c1a400a8561414bf65: Generated new TSK 0
I20260620 18:49:30.952828 4209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:30.967669 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4120
W20260620 18:49:30.968782 4182 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50197
W20260620 18:49:30.969558 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4120: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:30.969697 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4120
2026-06-20T18:49:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:31 dist-test-slave-zfp8 krb5kdc[4218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:31 dist-test-slave-zfp8 krb5kdc[4218](info): set up 2 sockets
Jun 20 18:49:31 dist-test-slave-zfp8 krb5kdc[4218](info): commencing operation
krb5kdc: starting...
W20260620 18:49:32.070036 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:32 dist-test-slave-zfp8 krb5kdc[4218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:32.332515 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:32.665458 4231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:32.665742 4231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:32.665915 4231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:32.673704 4231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:32.673815 4231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:32.673903 4231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:32.673995 4231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:32.674073 4231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:32.683956 4231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:32.685258 4231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:32.686604 4231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:32.698774 4239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:32.698843 4236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:32.698839 4237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:32.699577 4231 server_base.cc:1034] running on GCE node
I20260620 18:49:32.700417 4231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:32.701812 4231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:32.703042 4231 hybrid_clock.cc:648] HybridClock initialized: now 1781981372702975 us; error 76 us; skew 500 ppm
I20260620 18:49:32.703452 4231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:32 dist-test-slave-zfp8 krb5kdc[4218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:32.706439 4231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:32.707923 4231 webserver.cc:460] Webserver started at http://127.0.0.1:43661/ using document root <none> and password file <none>
I20260620 18:49:32.708639 4231 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:32.708778 4231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:32.709080 4231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:32.712116 4231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40"
format_stamp: "Formatted at 2026-06-20 18:49:32 on dist-test-slave-zfp8"
server_key: "215f25be9be2c779873a7a8e0ad2b262"
server_key_iv: "585f966d0d8cf3210336ab430a2027db"
server_key_version: "encryptionkey@0"
I20260620 18:49:32.712898 4231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40"
format_stamp: "Formatted at 2026-06-20 18:49:32 on dist-test-slave-zfp8"
server_key: "215f25be9be2c779873a7a8e0ad2b262"
server_key_iv: "585f966d0d8cf3210336ab430a2027db"
server_key_version: "encryptionkey@0"
I20260620 18:49:32.717943 4231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:49:32.721364 4246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:32.722604 4231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:49:32.722790 4231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40"
format_stamp: "Formatted at 2026-06-20 18:49:32 on dist-test-slave-zfp8"
server_key: "215f25be9be2c779873a7a8e0ad2b262"
server_key_iv: "585f966d0d8cf3210336ab430a2027db"
server_key_version: "encryptionkey@0"
I20260620 18:49:32.722991 4231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:32.728847 4231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:32.732932 4231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:32.733223 4231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:32.751492 4231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40329
I20260620 18:49:32.751536 4297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40329 every 8 connection(s)
I20260620 18:49:32.752899 4231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:32.755878 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4231
I20260620 18:49:32.756073 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:32.756507 15771 external_mini_cluster.cc:1428] Setting key 0b750f94b1c8ed53ad1050a420f89848
I20260620 18:49:32.756644 4298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:32.763763 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:32.758073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54051 (local address 127.0.0.1:40329)
0620 18:49:32.758722 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:32.758733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:32.758772 (+ 39us) server_negotiation.cc:407] Connection header received
0620 18:49:32.760391 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:32.760412 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:32.760749 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:32.761150 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:32.762151 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:32.762157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:32.762172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:32.762185 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:32.762252 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:32.762280 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:32.762509 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:32.762645 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 18:49:32.762825 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":269,"threads_started":1}
I20260620 18:49:32.767158 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:32.769356 4298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40: Bootstrap starting.
I20260620 18:49:32.772820 4298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:32.773942 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:32.771572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33717 (local address 127.0.0.1:40329)
0620 18:49:32.771751 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:32.771757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:32.772163 (+ 406us) server_negotiation.cc:407] Connection header received
0620 18:49:32.772295 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:32.772299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:32.772356 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:32.772466 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:32.773320 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:32.773325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:32.773329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:32.773347 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:32.773392 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:32.773411 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:32.773522 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:32.773588 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 18:49:32.773641 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 18:49:32.774233 4298 log.cc:824] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:32.776991 4298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40: No bootstrap required, opened a new log
I20260620 18:49:32.783948 4298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260620 18:49:32.784307 4298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:32.784423 4298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a2c0bf9fc8474bacf3b3bbe42c4c40, State: Initialized, Role: FOLLOWER
I20260620 18:49:32.784976 4298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260620 18:49:32.785202 4298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:32.785352 4298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:32.785517 4298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:32.787741 4298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260620 18:49:32.788221 4298 leader_election.cc:302] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56a2c0bf9fc8474bacf3b3bbe42c4c40; no voters:
I20260620 18:49:32.788820 4298 leader_election.cc:288] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:32.788966 4308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:32.789644 4308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [term 1 LEADER]: Becoming Leader. State: Replica: 56a2c0bf9fc8474bacf3b3bbe42c4c40, State: Running, Role: LEADER
I20260620 18:49:32.790169 4308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260620 18:49:32.790911 4298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:32.794339 4309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } } }
I20260620 18:49:32.794426 4310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56a2c0bf9fc8474bacf3b3bbe42c4c40. Latest consensus state: current_term: 1 leader_uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56a2c0bf9fc8474bacf3b3bbe42c4c40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } } }
I20260620 18:49:32.794736 4309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:32.794821 4310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:32.796553 4314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:32.802076 4314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:32.811028 4314 catalog_manager.cc:1347] Generated new cluster ID: 61fbb72f9db04014b8e9e44a818aca6f
I20260620 18:49:32.811151 4314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:32.836870 4314 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:32.837406 4314 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:32.845230 4314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56a2c0bf9fc8474bacf3b3bbe42c4c40: Generated new TSK 0
I20260620 18:49:32.846280 4314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:32.855376 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4231
W20260620 18:49:32.857036 4293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54051
W20260620 18:49:32.857788 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4231: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:32.857942 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4231
2026-06-20T18:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:32 dist-test-slave-zfp8 krb5kdc[4329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:32 dist-test-slave-zfp8 krb5kdc[4329](info): set up 2 sockets
Jun 20 18:49:32 dist-test-slave-zfp8 krb5kdc[4329](info): commencing operation
krb5kdc: starting...
W20260620 18:49:33.963704 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:34 dist-test-slave-zfp8 krb5kdc[4329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:34.218605 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:34.556543 4342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:34.556859 4342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:34.557016 4342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:34.564620 4342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:34.564836 4342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:34.564934 4342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:34.565043 4342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:34.565112 4342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:34.575122 4342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:34.576505 4342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:34.577838 4342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:34.589445 4347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:34.589399 4350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:34.589352 4348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:34.590304 4342 server_base.cc:1034] running on GCE node
I20260620 18:49:34.591112 4342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:34.592533 4342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:34.593757 4342 hybrid_clock.cc:648] HybridClock initialized: now 1781981374593719 us; error 27 us; skew 500 ppm
I20260620 18:49:34.594133 4342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:34 dist-test-slave-zfp8 krb5kdc[4329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:34.597201 4342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:34.598587 4342 webserver.cc:460] Webserver started at http://127.0.0.1:42679/ using document root <none> and password file <none>
I20260620 18:49:34.599305 4342 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:34.599447 4342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:34.599741 4342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:34.602737 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b"
format_stamp: "Formatted at 2026-06-20 18:49:34 on dist-test-slave-zfp8"
server_key: "da19b809c0ba3d610bbed398a1c2b5c3"
server_key_iv: "e01cffc08f480e78d9a3204ff3d04623"
server_key_version: "encryptionkey@0"
I20260620 18:49:34.603559 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b"
format_stamp: "Formatted at 2026-06-20 18:49:34 on dist-test-slave-zfp8"
server_key: "da19b809c0ba3d610bbed398a1c2b5c3"
server_key_iv: "e01cffc08f480e78d9a3204ff3d04623"
server_key_version: "encryptionkey@0"
I20260620 18:49:34.608551 4342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:49:34.611917 4357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:34.613163 4342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 18:49:34.613338 4342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b"
format_stamp: "Formatted at 2026-06-20 18:49:34 on dist-test-slave-zfp8"
server_key: "da19b809c0ba3d610bbed398a1c2b5c3"
server_key_iv: "e01cffc08f480e78d9a3204ff3d04623"
server_key_version: "encryptionkey@0"
I20260620 18:49:34.613526 4342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:34.659006 4342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:34.663599 4342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:34.663899 4342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:34.681847 4342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37795
I20260620 18:49:34.681875 4408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37795 every 8 connection(s)
I20260620 18:49:34.683224 4342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:34.684564 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4342
I20260620 18:49:34.684763 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:34.685292 15771 external_mini_cluster.cc:1428] Setting key f0339223ea90174b2194f9b28be89fe9
I20260620 18:49:34.687922 4409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:49:34 dist-test-slave-zfp8 krb5kdc[4329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:49:34.701511 4409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b: Bootstrap starting.
I20260620 18:49:34.704918 4409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:34.705968 4409 log.cc:824] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:34.708950 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:34.687119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:37407 (local address 127.0.0.1:37795)
0620 18:49:34.687969 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:34.688002 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:34.688052 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:49:34.689688 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:34.689709 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:34.690086 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:34.690423 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:34.692234 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:34.693688 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:34.695549 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:34.696208 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:34.696348 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:34.701233 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:34.701245 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:34.701266 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:34.701318 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:34.704511 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:34.705417 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:34.705424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:34.705431 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:34.705600 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:34.706079 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:34.706082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:34.706084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:34.706825 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:34.707194 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:34.707617 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 18:49:34.707941 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":353,"threads_started":1}
I20260620 18:49:34.709158 4409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b: No bootstrap required, opened a new log
I20260620 18:49:34.712756 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:34.719412 4409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37795 } }
I20260620 18:49:34.719815 4409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:34.719914 4409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6bfdc98dec47b8a05ea8cb7ab1142b, State: Initialized, Role: FOLLOWER
I20260620 18:49:34.720564 4409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37795 } }
I20260620 18:49:34.720842 4409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:34.721006 4409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:34.721185 4409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:34.724045 4409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37795 } }
I20260620 18:49:34.724602 4409 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6bfdc98dec47b8a05ea8cb7ab1142b; no voters:
I20260620 18:49:34.725260 4409 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:34.725421 4419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:34.727389 4409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:34.727634 4419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [term 1 LEADER]: Becoming Leader. State: Replica: 3d6bfdc98dec47b8a05ea8cb7ab1142b, State: Running, Role: LEADER
I20260620 18:49:34.728271 4419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37795 } }
I20260620 18:49:34.732519 4420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37795 } } }
I20260620 18:49:34.732908 4420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:34.733368 4421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6bfdc98dec47b8a05ea8cb7ab1142b. Latest consensus state: current_term: 1 leader_uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6bfdc98dec47b8a05ea8cb7ab1142b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37795 } } }
I20260620 18:49:34.733693 4421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:34.737794 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:34.718196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:59599 (local address 127.0.0.1:37795)
0620 18:49:34.718397 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:34.718404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:34.718813 (+ 409us) server_negotiation.cc:407] Connection header received
0620 18:49:34.718953 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:34.718959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:34.719041 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:34.719169 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:34.720775 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:34.721959 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:34.723174 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:34.723865 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:34.724012 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:34.727393 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:34.727402 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:34.727410 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:34.727455 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:34.731972 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:34.733284 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:34.733293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:34.733300 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:34.733621 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:34.734362 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:34.734369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:34.734375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:34.734922 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:34.735316 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:34.736995 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0620 18:49:34.737273 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 18:49:34.742533 4425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:34.748284 4425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:34.756966 4425 catalog_manager.cc:1347] Generated new cluster ID: 7ee6043c519c482c843929a5a6171a15
I20260620 18:49:34.757094 4425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:34.816753 4425 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:34.818199 4425 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:34.834219 4425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d6bfdc98dec47b8a05ea8cb7ab1142b: Generated new TSK 0
I20260620 18:49:34.835081 4425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:34.847630 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4342
I20260620 18:49:34.848784 4404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:34.947096 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4342
W20260620 18:49:34.964828 4327 connection.cc:535] client connection to 127.0.0.1:37795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37795): Connection reset by peer (error 104)
2026-06-20T18:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2089 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:35 dist-test-slave-zfp8 krb5kdc[4442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:35 dist-test-slave-zfp8 krb5kdc[4442](info): set up 2 sockets
Jun 20 18:49:35 dist-test-slave-zfp8 krb5kdc[4442](info): commencing operation
krb5kdc: starting...
W20260620 18:49:36.043627 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:36 dist-test-slave-zfp8 krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:36.287403 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:36.622222 4455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:36.622512 4455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:36.622605 4455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:36.629587 4455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:36.629701 4455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:36.629755 4455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:36.629807 4455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:36.629853 4455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:36.639524 4455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:36.640848 4455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:36.642149 4455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:36.653793 4460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:36.653748 4463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:36.653764 4461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:36.654632 4455 server_base.cc:1034] running on GCE node
I20260620 18:49:36.655445 4455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:36.656813 4455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:36.658047 4455 hybrid_clock.cc:648] HybridClock initialized: now 1781981376658017 us; error 44 us; skew 500 ppm
I20260620 18:49:36.658390 4455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:36 dist-test-slave-zfp8 krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:36.661630 4455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:36.663084 4455 webserver.cc:460] Webserver started at http://127.0.0.1:37737/ using document root <none> and password file <none>
I20260620 18:49:36.663792 4455 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:36.663929 4455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:36.664240 4455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:36.667176 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "4312fb66f0b647dc83bed5d1c26b6498"
format_stamp: "Formatted at 2026-06-20 18:49:36 on dist-test-slave-zfp8"
server_key: "8777f4bc521110a8a34e427231ad3c23"
server_key_iv: "68929871ee2812bfd025ab22165d7886"
server_key_version: "encryptionkey@0"
I20260620 18:49:36.667994 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "4312fb66f0b647dc83bed5d1c26b6498"
format_stamp: "Formatted at 2026-06-20 18:49:36 on dist-test-slave-zfp8"
server_key: "8777f4bc521110a8a34e427231ad3c23"
server_key_iv: "68929871ee2812bfd025ab22165d7886"
server_key_version: "encryptionkey@0"
I20260620 18:49:36.673228 4455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:49:36.676700 4470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:36.678012 4455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:49:36.678232 4455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "4312fb66f0b647dc83bed5d1c26b6498"
format_stamp: "Formatted at 2026-06-20 18:49:36 on dist-test-slave-zfp8"
server_key: "8777f4bc521110a8a34e427231ad3c23"
server_key_iv: "68929871ee2812bfd025ab22165d7886"
server_key_version: "encryptionkey@0"
I20260620 18:49:36.678431 4455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:36.702839 4455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:36.707300 4455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:36.707588 4455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:36.725991 4455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35833
I20260620 18:49:36.726008 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35833 every 8 connection(s)
I20260620 18:49:36.727667 4455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:36.731420 4522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:36.732651 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4455
I20260620 18:49:36.732887 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:36.733400 15771 external_mini_cluster.cc:1428] Setting key ad5dde96783b3a82896468581b871609
I20260620 18:49:36.745857 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498: Bootstrap starting.
I20260620 18:49:36.748060 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:36.735481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:59961 (local address 127.0.0.1:35833)
0620 18:49:36.736336 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:36.736356 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:36.736418 (+ 62us) server_negotiation.cc:407] Connection header received
0620 18:49:36.738047 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:36.738075 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:36.738613 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:36.739040 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:36.739618 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:36.741475 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:36.742813 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:36.743930 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:36.744113 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:36.745255 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:36.745265 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:36.745283 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:36.746009 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:36.746103 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:36.746135 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:36.746297 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:36.746578 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 18:49:36.746741 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":362,"threads_started":1}
I20260620 18:49:36.749675 4522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:36.750980 4522 log.cc:824] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:36.752158 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:36.753728 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498: No bootstrap required, opened a new log
I20260620 18:49:36.763082 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:36.756559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:39177 (local address 127.0.0.1:35833)
0620 18:49:36.756706 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:36.756710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:36.757023 (+ 313us) server_negotiation.cc:407] Connection header received
0620 18:49:36.757211 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:36.757215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:36.757275 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:36.757357 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:36.757937 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:36.758702 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:36.760610 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:36.761054 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:36.761163 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:36.762345 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:36.762350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:36.762353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:36.762369 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:36.762403 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:36.762417 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:36.762498 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:36.762706 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 18:49:36.762745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 18:49:36.763036 4522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4312fb66f0b647dc83bed5d1c26b6498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } }
I20260620 18:49:36.763532 4522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:36.763644 4522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4312fb66f0b647dc83bed5d1c26b6498, State: Initialized, Role: FOLLOWER
I20260620 18:49:36.764252 4522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4312fb66f0b647dc83bed5d1c26b6498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } }
I20260620 18:49:36.764492 4522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:36.764644 4522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:36.764807 4522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:36.766891 4522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4312fb66f0b647dc83bed5d1c26b6498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } }
I20260620 18:49:36.767441 4522 leader_election.cc:302] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4312fb66f0b647dc83bed5d1c26b6498; no voters:
I20260620 18:49:36.768029 4522 leader_election.cc:288] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:36.768172 4532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:36.769297 4532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [term 1 LEADER]: Becoming Leader. State: Replica: 4312fb66f0b647dc83bed5d1c26b6498, State: Running, Role: LEADER
I20260620 18:49:36.769783 4532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4312fb66f0b647dc83bed5d1c26b6498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } }
I20260620 18:49:36.770126 4522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:36.772598 4534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4312fb66f0b647dc83bed5d1c26b6498. Latest consensus state: current_term: 1 leader_uuid: "4312fb66f0b647dc83bed5d1c26b6498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4312fb66f0b647dc83bed5d1c26b6498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } } }
I20260620 18:49:36.772992 4534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:36.772715 4533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4312fb66f0b647dc83bed5d1c26b6498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4312fb66f0b647dc83bed5d1c26b6498" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35833 } } }
I20260620 18:49:36.773116 4533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:36.773895 4540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:36.779919 4540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:36.788913 4540 catalog_manager.cc:1347] Generated new cluster ID: 261fb786a463436f9f831a8f6e101cd7
I20260620 18:49:36.789024 4540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:36.836685 4540 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:36.838562 4540 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:36.851409 4540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4312fb66f0b647dc83bed5d1c26b6498: Generated new TSK 0
I20260620 18:49:36.852285 4540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:36.859359 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4455
W20260620 18:49:36.860775 4517 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.66:59961
W20260620 18:49:36.861578 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4455: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:36.861737 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4455
W20260620 18:49:36.878716 4440 connection.cc:535] client connection to 127.0.0.1:35833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35833): Connection reset by peer (error 104)
2026-06-20T18:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:36 dist-test-slave-zfp8 krb5kdc[4553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:36 dist-test-slave-zfp8 krb5kdc[4553](info): set up 2 sockets
Jun 20 18:49:36 dist-test-slave-zfp8 krb5kdc[4553](info): commencing operation
krb5kdc: starting...
W20260620 18:49:37.975697 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:38 dist-test-slave-zfp8 krb5kdc[4553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:38.226701 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:38.560823 4566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:38.561110 4566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:38.561267 4566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:38.568394 4566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:38.568506 4566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:38.568593 4566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:38.568676 4566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:38.568742 4566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:38.578436 4566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:38.579811 4566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:38.581110 4566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:38.593010 4572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:38.593143 4571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:38.593139 4574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:38.595273 4566 server_base.cc:1034] running on GCE node
I20260620 18:49:38.596092 4566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:38.597474 4566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:38.598681 4566 hybrid_clock.cc:648] HybridClock initialized: now 1781981378598643 us; error 27 us; skew 500 ppm
I20260620 18:49:38.599032 4566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:38 dist-test-slave-zfp8 krb5kdc[4553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:38.607502 4566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:38.608989 4566 webserver.cc:460] Webserver started at http://127.0.0.1:41743/ using document root <none> and password file <none>
I20260620 18:49:38.609618 4566 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:38.609717 4566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:38.609977 4566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:38.613021 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "48754c705d534f36820b6be7ed62b5b4"
format_stamp: "Formatted at 2026-06-20 18:49:38 on dist-test-slave-zfp8"
server_key: "b2d9783a26adaf236b90a72bf53b0e91"
server_key_iv: "6538816a45fe769f47a06787f524f98f"
server_key_version: "encryptionkey@0"
I20260620 18:49:38.613744 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "48754c705d534f36820b6be7ed62b5b4"
format_stamp: "Formatted at 2026-06-20 18:49:38 on dist-test-slave-zfp8"
server_key: "b2d9783a26adaf236b90a72bf53b0e91"
server_key_iv: "6538816a45fe769f47a06787f524f98f"
server_key_version: "encryptionkey@0"
I20260620 18:49:38.618818 4566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:49:38.622272 4581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:38.623696 4566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:38.623888 4566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "48754c705d534f36820b6be7ed62b5b4"
format_stamp: "Formatted at 2026-06-20 18:49:38 on dist-test-slave-zfp8"
server_key: "b2d9783a26adaf236b90a72bf53b0e91"
server_key_iv: "6538816a45fe769f47a06787f524f98f"
server_key_version: "encryptionkey@0"
I20260620 18:49:38.624085 4566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:38.629864 4566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:38.634063 4566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:38.634361 4566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:38.652498 4566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36857
I20260620 18:49:38.652539 4632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36857 every 8 connection(s)
I20260620 18:49:38.653927 4566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:38.657778 4633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:38.662220 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4566
I20260620 18:49:38.662441 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:38.662858 15771 external_mini_cluster.cc:1428] Setting key 98f352100c87850941ba8d01df1124bb
I20260620 18:49:38.670179 4633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4: Bootstrap starting.
I20260620 18:49:38.672273 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:38.664658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:54017 (local address 127.0.0.1:36857)
0620 18:49:38.665488 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:38.665508 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:38.665564 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:49:38.668306 (+ 2742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:38.668335 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:38.668799 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:38.669252 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:38.670232 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:38.670239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:38.670256 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:38.670276 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:38.670345 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:38.670375 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:38.670602 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:38.670741 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 18:49:38.670923 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":325,"threads_started":1}
I20260620 18:49:38.673874 4633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:38.675065 4633 log.cc:824] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:38.676075 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:38.678229 4633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4: No bootstrap required, opened a new log
I20260620 18:49:38.683897 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:38.681425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:55565 (local address 127.0.0.1:36857)
0620 18:49:38.681606 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:38.681612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:38.681972 (+ 360us) server_negotiation.cc:407] Connection header received
0620 18:49:38.682112 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:38.682118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:38.682180 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:38.682283 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:38.683226 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:38.683256 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:38.683261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:38.683281 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:38.683326 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:38.683346 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:38.683477 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:38.683533 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 18:49:38.683593 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 18:49:38.687501 4633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48754c705d534f36820b6be7ed62b5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } }
I20260620 18:49:38.687835 4633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:38.687907 4633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48754c705d534f36820b6be7ed62b5b4, State: Initialized, Role: FOLLOWER
I20260620 18:49:38.688385 4633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48754c705d534f36820b6be7ed62b5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } }
I20260620 18:49:38.688580 4633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:38.688690 4633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:38.688818 4633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:38.690898 4633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48754c705d534f36820b6be7ed62b5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } }
I20260620 18:49:38.691452 4633 leader_election.cc:302] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48754c705d534f36820b6be7ed62b5b4; no voters:
I20260620 18:49:38.691974 4633 leader_election.cc:288] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:38.692121 4643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:38.693207 4643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [term 1 LEADER]: Becoming Leader. State: Replica: 48754c705d534f36820b6be7ed62b5b4, State: Running, Role: LEADER
I20260620 18:49:38.693907 4643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48754c705d534f36820b6be7ed62b5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } }
I20260620 18:49:38.694216 4633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:38.696702 4645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48754c705d534f36820b6be7ed62b5b4. Latest consensus state: current_term: 1 leader_uuid: "48754c705d534f36820b6be7ed62b5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48754c705d534f36820b6be7ed62b5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } } }
I20260620 18:49:38.697165 4645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:38.696739 4644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48754c705d534f36820b6be7ed62b5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48754c705d534f36820b6be7ed62b5b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } } }
I20260620 18:49:38.697523 4644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:38.699504 4651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:38.704636 4651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:38.713075 4651 catalog_manager.cc:1347] Generated new cluster ID: b330fadc85a848869f72884c8933c76a
I20260620 18:49:38.713189 4651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:38.731590 4651 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:38.732039 4651 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:38.746049 4651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48754c705d534f36820b6be7ed62b5b4: Generated new TSK 0
I20260620 18:49:38.746917 4651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:38.753207 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4566
W20260620 18:49:38.754336 4628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.66:54017
W20260620 18:49:38.755110 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:38.755275 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4566
2026-06-20T18:49:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:38 dist-test-slave-zfp8 krb5kdc[4664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:38 dist-test-slave-zfp8 krb5kdc[4664](info): set up 2 sockets
Jun 20 18:49:38 dist-test-slave-zfp8 krb5kdc[4664](info): commencing operation
krb5kdc: starting...
W20260620 18:49:39.863787 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:40 dist-test-slave-zfp8 krb5kdc[4664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:40.120290 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:40.449978 4677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:40.450327 4677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:40.450515 4677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:40.459105 4677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:40.459283 4677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:40.459394 4677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:40.459491 4677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:40.459555 4677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:40.469695 4677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:40.471072 4677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:40.472504 4677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:40.484100 4685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:40.484170 4682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:40.484150 4683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:40.485332 4677 server_base.cc:1034] running on GCE node
I20260620 18:49:40.486066 4677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:40.487463 4677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:40.488699 4677 hybrid_clock.cc:648] HybridClock initialized: now 1781981380488662 us; error 41 us; skew 500 ppm
I20260620 18:49:40.489068 4677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:40 dist-test-slave-zfp8 krb5kdc[4664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:40.492312 4677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:40.493952 4677 webserver.cc:460] Webserver started at http://127.0.0.1:44561/ using document root <none> and password file <none>
I20260620 18:49:40.494655 4677 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:40.494807 4677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:40.495108 4677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:40.498172 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "273f98538a654e7aa7b8f5f078ea1e1f"
format_stamp: "Formatted at 2026-06-20 18:49:40 on dist-test-slave-zfp8"
server_key: "c653af9c3d1c9204ed9208eb91f134d9"
server_key_iv: "ce007de27bab024589c90dc8d965e8bc"
server_key_version: "encryptionkey@0"
I20260620 18:49:40.498979 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "273f98538a654e7aa7b8f5f078ea1e1f"
format_stamp: "Formatted at 2026-06-20 18:49:40 on dist-test-slave-zfp8"
server_key: "c653af9c3d1c9204ed9208eb91f134d9"
server_key_iv: "ce007de27bab024589c90dc8d965e8bc"
server_key_version: "encryptionkey@0"
I20260620 18:49:40.504388 4677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:49:40.507949 4692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:40.509347 4677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:40.509541 4677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "273f98538a654e7aa7b8f5f078ea1e1f"
format_stamp: "Formatted at 2026-06-20 18:49:40 on dist-test-slave-zfp8"
server_key: "c653af9c3d1c9204ed9208eb91f134d9"
server_key_iv: "ce007de27bab024589c90dc8d965e8bc"
server_key_version: "encryptionkey@0"
I20260620 18:49:40.509748 4677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:40.534145 4677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:40.538383 4677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:40.538623 4677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:40.557878 4677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42339
I20260620 18:49:40.557928 4743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42339 every 8 connection(s)
I20260620 18:49:40.559399 4677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:40.563447 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:40.565137 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4677
I20260620 18:49:40.565294 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:40.565753 15771 external_mini_cluster.cc:1428] Setting key ec7985b61736b82ec7b822c1bbdb1ef3
I20260620 18:49:40.577095 4744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f: Bootstrap starting.
Jun 20 18:49:40 dist-test-slave-zfp8 krb5kdc[4664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 18:49:40.580730 4744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:40.581980 4744 log.cc:824] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:40.585415 4744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f: No bootstrap required, opened a new log
I20260620 18:49:40.588840 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:40.567673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:47421 (local address 127.0.0.1:42339)
0620 18:49:40.568451 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:40.568463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:40.568506 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:49:40.570195 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:40.570222 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:40.570682 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:40.571104 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:40.572636 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:40.573983 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:40.575411 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:40.576108 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:40.576311 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:40.581036 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:40.581044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:40.581059 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:40.581099 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:40.584452 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:40.585415 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:40.585421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:40.585427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:40.585627 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:40.586119 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:40.586123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:40.586125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:40.586635 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:40.586885 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:40.587458 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 18:49:40.587812 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":331,"threads_started":1}
I20260620 18:49:40.592906 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:40.594416 4744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f98538a654e7aa7b8f5f078ea1e1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } }
I20260620 18:49:40.594838 4744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:40.594938 4744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273f98538a654e7aa7b8f5f078ea1e1f, State: Initialized, Role: FOLLOWER
I20260620 18:49:40.595598 4744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f98538a654e7aa7b8f5f078ea1e1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } }
I20260620 18:49:40.595885 4744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:40.596026 4744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:40.596247 4744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:40.599306 4744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f98538a654e7aa7b8f5f078ea1e1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } }
I20260620 18:49:40.599839 4744 leader_election.cc:302] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273f98538a654e7aa7b8f5f078ea1e1f; no voters:
I20260620 18:49:40.600442 4744 leader_election.cc:288] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:40.600591 4754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:40.601905 4754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [term 1 LEADER]: Becoming Leader. State: Replica: 273f98538a654e7aa7b8f5f078ea1e1f, State: Running, Role: LEADER
I20260620 18:49:40.602576 4754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f98538a654e7aa7b8f5f078ea1e1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } }
I20260620 18:49:40.602857 4744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:40.605340 4755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273f98538a654e7aa7b8f5f078ea1e1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f98538a654e7aa7b8f5f078ea1e1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } } }
I20260620 18:49:40.605742 4755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:40.608001 4756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273f98538a654e7aa7b8f5f078ea1e1f. Latest consensus state: current_term: 1 leader_uuid: "273f98538a654e7aa7b8f5f078ea1e1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f98538a654e7aa7b8f5f078ea1e1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } } }
I20260620 18:49:40.608404 4756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:40.609579 4759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:40.615202 4759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:40.620952 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:40.598188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:36861 (local address 127.0.0.1:42339)
0620 18:49:40.598386 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:40.598393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:40.598766 (+ 373us) server_negotiation.cc:407] Connection header received
0620 18:49:40.598902 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:40.598908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:40.598989 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:40.599102 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:40.601026 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:40.602457 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:40.604620 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:40.605630 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:40.605775 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:40.609987 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:40.609997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:40.610005 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:40.610061 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:40.615265 (+ 5204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:40.616605 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:40.616613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:40.616619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:40.616815 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:40.617548 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:40.617556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:40.617562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:40.618131 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:40.619934 (+ 1803us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:40.620169 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:49:40.620409 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:49:40.631026 4759 catalog_manager.cc:1347] Generated new cluster ID: de163dd7cfb942bc829e164cb5808d7a
I20260620 18:49:40.631184 4759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:40.647853 4759 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:40.649713 4759 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:40.658733 4759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273f98538a654e7aa7b8f5f078ea1e1f: Generated new TSK 0
I20260620 18:49:40.659672 4759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:40.669939 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4677
I20260620 18:49:40.671161 4739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:40.762504 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4677
W20260620 18:49:40.779960 4662 connection.cc:535] client connection to 127.0.0.1:42339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42339): Connection reset by peer (error 104)
2026-06-20T18:49:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2007 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:40 dist-test-slave-zfp8 krb5kdc[4776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:40 dist-test-slave-zfp8 krb5kdc[4776](info): set up 2 sockets
Jun 20 18:49:40 dist-test-slave-zfp8 krb5kdc[4776](info): commencing operation
krb5kdc: starting...
W20260620 18:49:41.875869 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:42 dist-test-slave-zfp8 krb5kdc[4776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:42.123356 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:42.482546 4789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:42.482905 4789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:42.483106 4789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:42.490552 4789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:42.490705 4789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:42.490829 4789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:42.490952 4789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:42.491047 4789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:42.501380 4789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:42.502771 4789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:42.504251 4789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:42.516835 4797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:42.516950 4794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:42.516983 4795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:42.517834 4789 server_base.cc:1034] running on GCE node
I20260620 18:49:42.518618 4789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:42.520215 4789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:42.521422 4789 hybrid_clock.cc:648] HybridClock initialized: now 1781981382521374 us; error 44 us; skew 500 ppm
I20260620 18:49:42.521775 4789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:42 dist-test-slave-zfp8 krb5kdc[4776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:42.524951 4789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:42.526304 4789 webserver.cc:460] Webserver started at http://127.0.0.1:38763/ using document root <none> and password file <none>
I20260620 18:49:42.526924 4789 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:42.527020 4789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:42.527359 4789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:42.530249 4789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "4fb8daf490104bdea8af446b68a15a0c"
format_stamp: "Formatted at 2026-06-20 18:49:42 on dist-test-slave-zfp8"
server_key: "35aceedf86812dcfe34865d02bced293"
server_key_iv: "35fd8470099881ee90dfc551e3e73301"
server_key_version: "encryptionkey@0"
I20260620 18:49:42.530941 4789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "4fb8daf490104bdea8af446b68a15a0c"
format_stamp: "Formatted at 2026-06-20 18:49:42 on dist-test-slave-zfp8"
server_key: "35aceedf86812dcfe34865d02bced293"
server_key_iv: "35fd8470099881ee90dfc551e3e73301"
server_key_version: "encryptionkey@0"
I20260620 18:49:42.536468 4789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:49:42.539770 4804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:42.541105 4789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 18:49:42.541291 4789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "4fb8daf490104bdea8af446b68a15a0c"
format_stamp: "Formatted at 2026-06-20 18:49:42 on dist-test-slave-zfp8"
server_key: "35aceedf86812dcfe34865d02bced293"
server_key_iv: "35fd8470099881ee90dfc551e3e73301"
server_key_version: "encryptionkey@0"
I20260620 18:49:42.541481 4789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:42.560643 4789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:42.565332 4789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:42.565644 4789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:42.584523 4789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44833
I20260620 18:49:42.584551 4855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44833 every 8 connection(s)
I20260620 18:49:42.586014 4789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:42.589810 4856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:42.593142 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4789
I20260620 18:49:42.593317 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:42.593850 15771 external_mini_cluster.cc:1428] Setting key 1f86c4f5acab07e5c9624ffa01e4f8b9
I20260620 18:49:42.601698 4856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c: Bootstrap starting.
I20260620 18:49:42.604947 4856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:42.606040 4856 log.cc:824] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:42.607677 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:42.595814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:49141 (local address 127.0.0.1:44833)
0620 18:49:42.596569 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:42.596583 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:42.596625 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:49:42.598307 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:42.598331 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:42.598755 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:42.599161 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:42.599777 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:42.601175 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:42.602611 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:42.603469 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:42.603689 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:42.604972 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:42.604979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:42.604994 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:42.605609 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:42.605683 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:42.605707 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:42.605880 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:42.606163 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:49:42.606335 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":332,"threads_started":1}
I20260620 18:49:42.609344 4856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c: No bootstrap required, opened a new log
I20260620 18:49:42.611815 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:42.617851 4856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb8daf490104bdea8af446b68a15a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260620 18:49:42.618244 4856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:42.618355 4856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb8daf490104bdea8af446b68a15a0c, State: Initialized, Role: FOLLOWER
I20260620 18:49:42.618985 4856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb8daf490104bdea8af446b68a15a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260620 18:49:42.619277 4856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:42.619462 4856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:42.619634 4856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:42.621940 4856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb8daf490104bdea8af446b68a15a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260620 18:49:42.622437 4856 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fb8daf490104bdea8af446b68a15a0c; no voters:
I20260620 18:49:42.623121 4856 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:42.623219 4866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:42.623626 4859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:42.617185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:43809 (local address 127.0.0.1:44833)
0620 18:49:42.617379 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:42.617385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:42.617672 (+ 287us) server_negotiation.cc:407] Connection header received
0620 18:49:42.617890 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:42.617894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:42.617961 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:42.618065 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:42.618639 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:42.619586 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:42.620998 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:42.621452 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:42.621585 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:42.622732 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:42.622738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:42.622745 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:42.622767 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:42.622831 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:42.622845 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:42.622989 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:42.623182 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 18:49:42.623261 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 18:49:42.624130 4866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [term 1 LEADER]: Becoming Leader. State: Replica: 4fb8daf490104bdea8af446b68a15a0c, State: Running, Role: LEADER
I20260620 18:49:42.624927 4866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb8daf490104bdea8af446b68a15a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260620 18:49:42.626904 4856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:42.628036 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb8daf490104bdea8af446b68a15a0c. Latest consensus state: current_term: 1 leader_uuid: "4fb8daf490104bdea8af446b68a15a0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb8daf490104bdea8af446b68a15a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } } }
I20260620 18:49:42.628088 4868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fb8daf490104bdea8af446b68a15a0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb8daf490104bdea8af446b68a15a0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } } }
I20260620 18:49:42.628423 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:42.628435 4868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:42.638500 4871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:42.642781 4871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:42.651373 4871 catalog_manager.cc:1347] Generated new cluster ID: 8372c5da4c6a4560a6188ed346aa52d4
I20260620 18:49:42.651543 4871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:42.663192 4871 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:42.664523 4871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:42.673889 4871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fb8daf490104bdea8af446b68a15a0c: Generated new TSK 0
I20260620 18:49:42.674738 4871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:42.686597 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4789
W20260620 18:49:42.688334 4851 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.66:49141
W20260620 18:49:42.689354 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4789: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:42.689524 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4789
W20260620 18:49:42.707041 4774 connection.cc:535] client connection to 127.0.0.1:44833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44833): Connection reset by peer (error 104)
2026-06-20T18:49:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:42 dist-test-slave-zfp8 krb5kdc[4887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:42 dist-test-slave-zfp8 krb5kdc[4887](info): set up 2 sockets
Jun 20 18:49:42 dist-test-slave-zfp8 krb5kdc[4887](info): commencing operation
krb5kdc: starting...
W20260620 18:49:43.799692 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:43 dist-test-slave-zfp8 krb5kdc[4887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 18:49:44.046141 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:44.371363 4900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:44.371676 4900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:44.371774 4900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:44.378994 4900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:44.379108 4900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:44.379155 4900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:44.379215 4900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:44.379323 4900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:44.389210 4900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36923
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:44.390566 4900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:44.391891 4900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:44.403682 4905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:44.403646 4906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:44.403590 4908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:44.408627 4900 server_base.cc:1034] running on GCE node
I20260620 18:49:44.409436 4900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:44.410827 4900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:44.412068 4900 hybrid_clock.cc:648] HybridClock initialized: now 1781981384412035 us; error 46 us; skew 500 ppm
I20260620 18:49:44.412406 4900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:44 dist-test-slave-zfp8 krb5kdc[4887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:44.419991 4900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 18:49:44.421530 4900 webserver.cc:460] Webserver started at http://127.0.0.1:44987/ using document root <none> and password file <none>
I20260620 18:49:44.422212 4900 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:44.422369 4900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:44.422664 4900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:44.425756 4900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "9acf6f5be4cd4400b21b581e8462d70d"
format_stamp: "Formatted at 2026-06-20 18:49:44 on dist-test-slave-zfp8"
server_key: "5092e8f6f93855330290c487b7cd1476"
server_key_iv: "e791b0cdbbca9493abdb5e5c0cac8a78"
server_key_version: "encryptionkey@0"
I20260620 18:49:44.426484 4900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "9acf6f5be4cd4400b21b581e8462d70d"
format_stamp: "Formatted at 2026-06-20 18:49:44 on dist-test-slave-zfp8"
server_key: "5092e8f6f93855330290c487b7cd1476"
server_key_iv: "e791b0cdbbca9493abdb5e5c0cac8a78"
server_key_version: "encryptionkey@0"
I20260620 18:49:44.431599 4900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 18:49:44.434928 4915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:44.436312 4900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:44.436488 4900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "9acf6f5be4cd4400b21b581e8462d70d"
format_stamp: "Formatted at 2026-06-20 18:49:44 on dist-test-slave-zfp8"
server_key: "5092e8f6f93855330290c487b7cd1476"
server_key_iv: "e791b0cdbbca9493abdb5e5c0cac8a78"
server_key_version: "encryptionkey@0"
I20260620 18:49:44.436695 4900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:44.442447 4900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:44.446704 4900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:44.446993 4900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:44.467159 4900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36923
I20260620 18:49:44.467204 4966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36923 every 8 connection(s)
I20260620 18:49:44.468600 4900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:44.469949 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 4900
I20260620 18:49:44.470119 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:44.470654 15771 external_mini_cluster.cc:1428] Setting key 7ab8c2dcd3127f1928baeead9de73e5c
I20260620 18:49:44.473253 4967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:44.478717 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:44.472625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:36891 (local address 127.0.0.1:36923)
0620 18:49:44.473435 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:44.473447 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:44.473490 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:49:44.475102 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:44.475122 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:44.475532 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:44.475867 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:44.476931 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:44.476937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:44.476949 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:44.476963 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:44.477042 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:44.477060 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:44.477236 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:44.477364 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 18:49:44.477574 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":354,"threads_started":1}
I20260620 18:49:44.482836 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:44.486531 4967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d: Bootstrap starting.
I20260620 18:49:44.490527 4967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:44.490633 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:44.488092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.66:58357 (local address 127.0.0.1:36923)
0620 18:49:44.488304 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:44.488313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:44.488790 (+ 477us) server_negotiation.cc:407] Connection header received
0620 18:49:44.488894 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:44.488899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:44.488964 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:44.489073 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:44.489946 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:44.489951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:44.489955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:44.489973 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:44.490020 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:44.490039 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:44.490172 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:44.490248 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 18:49:44.490308 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:49:44.491933 4967 log.cc:824] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:44.494568 4967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d: No bootstrap required, opened a new log
I20260620 18:49:44.502071 4967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acf6f5be4cd4400b21b581e8462d70d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260620 18:49:44.502471 4967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:44.502594 4967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acf6f5be4cd4400b21b581e8462d70d, State: Initialized, Role: FOLLOWER
I20260620 18:49:44.503178 4967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acf6f5be4cd4400b21b581e8462d70d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260620 18:49:44.503459 4967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:44.503611 4967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:44.503791 4967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:44.506170 4967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acf6f5be4cd4400b21b581e8462d70d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260620 18:49:44.506740 4967 leader_election.cc:302] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9acf6f5be4cd4400b21b581e8462d70d; no voters:
I20260620 18:49:44.507436 4967 leader_election.cc:288] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:44.507612 4977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:44.508368 4977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [term 1 LEADER]: Becoming Leader. State: Replica: 9acf6f5be4cd4400b21b581e8462d70d, State: Running, Role: LEADER
I20260620 18:49:44.508909 4977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acf6f5be4cd4400b21b581e8462d70d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260620 18:49:44.509682 4967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:44.511996 4978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9acf6f5be4cd4400b21b581e8462d70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acf6f5be4cd4400b21b581e8462d70d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } } }
I20260620 18:49:44.512521 4978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:44.513089 4979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9acf6f5be4cd4400b21b581e8462d70d. Latest consensus state: current_term: 1 leader_uuid: "9acf6f5be4cd4400b21b581e8462d70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9acf6f5be4cd4400b21b581e8462d70d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } } }
I20260620 18:49:44.513459 4979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:44.516347 4986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:44.522797 4986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:44.532650 4986 catalog_manager.cc:1347] Generated new cluster ID: fae7df5831d04903957e747f16fb7862
I20260620 18:49:44.532791 4986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:44.550648 4986 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:44.551196 4986 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:44.567271 4986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9acf6f5be4cd4400b21b581e8462d70d: Generated new TSK 0
I20260620 18:49:44.568408 4986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:44.576071 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4900
W20260620 18:49:44.577409 4962 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.66:36891
W20260620 18:49:44.578140 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 4900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:44.578292 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 4900
2026-06-20T18:49:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:44 dist-test-slave-zfp8 krb5kdc[4998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:44 dist-test-slave-zfp8 krb5kdc[4998](info): set up 2 sockets
Jun 20 18:49:44 dist-test-slave-zfp8 krb5kdc[4998](info): commencing operation
krb5kdc: starting...
W20260620 18:49:45.687808 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:45 dist-test-slave-zfp8 krb5kdc[4998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:45.936223 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35099
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:35099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:46.265260 5011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:46.265630 5011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:46.265797 5011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:46.273197 5011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:46.273363 5011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:46.273478 5011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:46.273592 5011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:46.273669 5011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:46.283805 5011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:35099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:46.285197 5011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:46.286597 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:46.298586 5017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:46.298627 5019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:46.298887 5016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:46.299594 5011 server_base.cc:1034] running on GCE node
I20260620 18:49:46.300305 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:46.301608 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:46.302809 5011 hybrid_clock.cc:648] HybridClock initialized: now 1781981386302778 us; error 30 us; skew 500 ppm
I20260620 18:49:46.303124 5011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:46 dist-test-slave-zfp8 krb5kdc[4998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981386, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:46.306138 5011 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:46.307554 5011 webserver.cc:460] Webserver started at http://127.15.102.254:41515/ using document root <none> and password file <none>
I20260620 18:49:46.308208 5011 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:46.308305 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:46.308560 5011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:46.311574 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "1a4814e0422441909692fd055b876ffc"
format_stamp: "Formatted at 2026-06-20 18:49:46 on dist-test-slave-zfp8"
server_key: "a86392cabef243f0fd7cf86f3cc58c39"
server_key_iv: "24906f39e0b54081a70b5fbe18f10179"
server_key_version: "encryptionkey@0"
I20260620 18:49:46.312317 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "1a4814e0422441909692fd055b876ffc"
format_stamp: "Formatted at 2026-06-20 18:49:46 on dist-test-slave-zfp8"
server_key: "a86392cabef243f0fd7cf86f3cc58c39"
server_key_iv: "24906f39e0b54081a70b5fbe18f10179"
server_key_version: "encryptionkey@0"
I20260620 18:49:46.317494 5011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 18:49:46.321007 5026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:46.322535 5011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:49:46.322685 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "1a4814e0422441909692fd055b876ffc"
format_stamp: "Formatted at 2026-06-20 18:49:46 on dist-test-slave-zfp8"
server_key: "a86392cabef243f0fd7cf86f3cc58c39"
server_key_iv: "24906f39e0b54081a70b5fbe18f10179"
server_key_version: "encryptionkey@0"
I20260620 18:49:46.322853 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:46.347050 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:46.350981 5011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:46.351207 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:46.369506 5011 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:35099
I20260620 18:49:46.369527 5077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:35099 every 8 connection(s)
I20260620 18:49:46.370926 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:46.374611 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:46.380338 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5011
I20260620 18:49:46.380543 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:46.381191 15771 external_mini_cluster.cc:1428] Setting key 8249b8e094d869dad756d24516efa613
I20260620 18:49:46.385919 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc: Bootstrap starting.
I20260620 18:49:46.389802 5078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:46.390851 5078 log.cc:824] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:46.393957 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc: No bootstrap required, opened a new log
Jun 20 18:49:46 dist-test-slave-zfp8 krb5kdc[4998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:46.402297 5078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4814e0422441909692fd055b876ffc" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35099 } }
I20260620 18:49:46.402715 5078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:46.402853 5078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4814e0422441909692fd055b876ffc, State: Initialized, Role: FOLLOWER
I20260620 18:49:46.403565 5078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4814e0422441909692fd055b876ffc" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35099 } }
I20260620 18:49:46.403826 5078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:46.403993 5078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:46.404191 5078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:46.405441 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:46.383131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42731 (local address 127.15.102.254:35099)
0620 18:49:46.384051 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:46.384063 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:46.384105 (+ 42us) server_negotiation.cc:407] Connection header received
0620 18:49:46.385769 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:46.385793 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:46.386263 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:46.386605 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:46.388571 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:46.389895 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:46.391270 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:46.392422 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:46.392623 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:46.396796 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:46.396803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:46.396817 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:46.396856 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:46.400552 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:46.401370 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:46.401376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:46.401386 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:46.401561 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:46.402189 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:46.402196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:46.402205 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:46.402789 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:46.403102 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:46.403647 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 18:49:46.404229 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":449,"threads_started":1}
I20260620 18:49:46.406899 5078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4814e0422441909692fd055b876ffc" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35099 } }
I20260620 18:49:46.407608 5078 leader_election.cc:302] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a4814e0422441909692fd055b876ffc; no voters:
I20260620 18:49:46.408425 5078 leader_election.cc:288] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:46.408614 5083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:46.409685 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:46.410542 5078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:46.413381 5083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [term 1 LEADER]: Becoming Leader. State: Replica: 1a4814e0422441909692fd055b876ffc, State: Running, Role: LEADER
I20260620 18:49:46.414077 5083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4814e0422441909692fd055b876ffc" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35099 } }
I20260620 18:49:46.420950 5085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4814e0422441909692fd055b876ffc. Latest consensus state: current_term: 1 leader_uuid: "1a4814e0422441909692fd055b876ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4814e0422441909692fd055b876ffc" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35099 } } }
I20260620 18:49:46.421483 5085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:46.422307 5084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a4814e0422441909692fd055b876ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4814e0422441909692fd055b876ffc" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 35099 } } }
I20260620 18:49:46.422652 5084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:46.436239 5102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:46.442011 5102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:46.445262 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:46.416790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40593 (local address 127.15.102.254:35099)
0620 18:49:46.416977 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:46.416986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:46.427088 (+ 10102us) server_negotiation.cc:407] Connection header received
0620 18:49:46.427695 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:46.427704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:46.427805 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:46.427892 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:46.430320 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:46.431852 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:46.434933 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:46.435519 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:46.435606 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:46.438578 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:46.438583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:46.438587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:46.438617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:46.442062 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:46.442947 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:46.442952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:46.442955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:46.443092 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:46.443651 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:46.443656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:46.443658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:46.443981 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:46.444274 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:46.444734 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 18:49:46.444940 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 18:49:46.452936 5102 catalog_manager.cc:1347] Generated new cluster ID: f42aae3ed01f444cbb0dcbebcd16fbe1
I20260620 18:49:46.453132 5102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:46.473347 5102 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:46.475296 5102 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:46.484416 5102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a4814e0422441909692fd055b876ffc: Generated new TSK 0
I20260620 18:49:46.485481 5102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:46.503026 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5011
I20260620 18:49:46.504426 5073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:46.596562 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5011
W20260620 18:49:46.613713 4996 connection.cc:535] client connection to 127.15.102.254:35099 recv error: Network error: failed to read from TLS socket (remote: 127.15.102.254:35099): Connection reset by peer (error 104)
2026-06-20T18:49:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2017 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:46 dist-test-slave-zfp8 krb5kdc[5111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:46 dist-test-slave-zfp8 krb5kdc[5111](info): set up 2 sockets
Jun 20 18:49:46 dist-test-slave-zfp8 krb5kdc[5111](info): commencing operation
krb5kdc: starting...
W20260620 18:49:47.703610 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:47 dist-test-slave-zfp8 krb5kdc[5111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:47.948644 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:43491
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:37903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:43491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:48.278590 5124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:48.278882 5124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:48.278977 5124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:48.286499 5124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:48.286624 5124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:48.286671 5124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:48.286725 5124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:48.286772 5124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:48.297127 5124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:43491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:43491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:48.298429 5124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:48.299758 5124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:48.311825 5132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:48.311852 5130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:48.311883 5129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:48.313002 5124 server_base.cc:1034] running on GCE node
I20260620 18:49:48.313762 5124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:48.315178 5124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:48.316432 5124 hybrid_clock.cc:648] HybridClock initialized: now 1781981388316395 us; error 31 us; skew 500 ppm
I20260620 18:49:48.316785 5124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:48 dist-test-slave-zfp8 krb5kdc[5111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981388, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:48.320029 5124 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:48.321468 5124 webserver.cc:460] Webserver started at http://127.15.102.254:35335/ using document root <none> and password file <none>
I20260620 18:49:48.322129 5124 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:48.322225 5124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:48.322466 5124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:48.325412 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "4a808d5faf7c4087a64ad7800e5c0e8b"
format_stamp: "Formatted at 2026-06-20 18:49:48 on dist-test-slave-zfp8"
server_key: "ea86d5a2477c9c0df67e70138d774836"
server_key_iv: "d210fe18c79e68bf7b2f3ebcf88df92a"
server_key_version: "encryptionkey@0"
I20260620 18:49:48.326246 5124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "4a808d5faf7c4087a64ad7800e5c0e8b"
format_stamp: "Formatted at 2026-06-20 18:49:48 on dist-test-slave-zfp8"
server_key: "ea86d5a2477c9c0df67e70138d774836"
server_key_iv: "d210fe18c79e68bf7b2f3ebcf88df92a"
server_key_version: "encryptionkey@0"
I20260620 18:49:48.331522 5124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260620 18:49:48.334962 5139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:48.336431 5124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:48.336611 5124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "4a808d5faf7c4087a64ad7800e5c0e8b"
format_stamp: "Formatted at 2026-06-20 18:49:48 on dist-test-slave-zfp8"
server_key: "ea86d5a2477c9c0df67e70138d774836"
server_key_iv: "d210fe18c79e68bf7b2f3ebcf88df92a"
server_key_version: "encryptionkey@0"
I20260620 18:49:48.336803 5124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:48.356575 5124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:48.360569 5124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:48.360853 5124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:48.379499 5124 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:43491
I20260620 18:49:48.379519 5190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:43491 every 8 connection(s)
I20260620 18:49:48.380860 5124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:48.383104 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5124
I20260620 18:49:48.383327 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:48.383728 15771 external_mini_cluster.cc:1428] Setting key c0acff886d56b627dc545a39a75d621c
I20260620 18:49:48.384817 5191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:48.398532 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:48.385444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60749 (local address 127.15.102.254:43491)
0620 18:49:48.386236 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:48.386250 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:48.386300 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:49:48.388310 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:48.388336 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:48.388847 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:48.389296 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:48.389780 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:48.391844 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:48.393312 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:48.394312 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:48.394493 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:48.395812 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:48.395823 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:48.395841 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:48.396600 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:48.396695 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:48.396730 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:48.396903 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:48.397077 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 18:49:48.397260 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":332,"threads_started":1}
I20260620 18:49:48.399688 5191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b: Bootstrap starting.
I20260620 18:49:48.402431 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:48.403002 5191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:48.404255 5191 log.cc:824] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:48.407675 5191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b: No bootstrap required, opened a new log
I20260620 18:49:48.414476 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:48.407553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60863 (local address 127.15.102.254:43491)
0620 18:49:48.407764 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:48.407769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:48.408224 (+ 455us) server_negotiation.cc:407] Connection header received
0620 18:49:48.408371 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:48.408376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:48.408454 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:48.408571 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:48.409022 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:48.410199 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:48.411568 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:48.412245 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:48.412368 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:48.413444 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:48.413455 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:48.413459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:48.413479 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:48.413539 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:48.413560 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:48.413688 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:48.414086 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 18:49:48.414139 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:49:48.417260 5191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a808d5faf7c4087a64ad7800e5c0e8b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43491 } }
I20260620 18:49:48.417618 5191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:48.417695 5191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a808d5faf7c4087a64ad7800e5c0e8b, State: Initialized, Role: FOLLOWER
I20260620 18:49:48.418175 5191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a808d5faf7c4087a64ad7800e5c0e8b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43491 } }
I20260620 18:49:48.418363 5191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:48.418465 5191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:48.418593 5191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:48.420775 5191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a808d5faf7c4087a64ad7800e5c0e8b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43491 } }
I20260620 18:49:48.421207 5191 leader_election.cc:302] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a808d5faf7c4087a64ad7800e5c0e8b; no voters:
I20260620 18:49:48.421854 5191 leader_election.cc:288] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:48.421963 5201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:48.423070 5201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [term 1 LEADER]: Becoming Leader. State: Replica: 4a808d5faf7c4087a64ad7800e5c0e8b, State: Running, Role: LEADER
I20260620 18:49:48.423684 5201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a808d5faf7c4087a64ad7800e5c0e8b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43491 } }
I20260620 18:49:48.424369 5191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:48.426708 5202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a808d5faf7c4087a64ad7800e5c0e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a808d5faf7c4087a64ad7800e5c0e8b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43491 } } }
I20260620 18:49:48.426657 5203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a808d5faf7c4087a64ad7800e5c0e8b. Latest consensus state: current_term: 1 leader_uuid: "4a808d5faf7c4087a64ad7800e5c0e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a808d5faf7c4087a64ad7800e5c0e8b" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43491 } } }
I20260620 18:49:48.427091 5202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:48.427155 5203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:48.428138 5208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:48.433537 5208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:48.442153 5208 catalog_manager.cc:1347] Generated new cluster ID: fc8de207d8024745a8e4cddc74a7a22a
I20260620 18:49:48.442306 5208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:48.469477 5208 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:48.471038 5208 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:48.483026 5208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a808d5faf7c4087a64ad7800e5c0e8b: Generated new TSK 0
I20260620 18:49:48.483875 5208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:48.489539 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5124
W20260620 18:49:48.490650 5186 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60749
W20260620 18:49:48.491438 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5124: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:48.491613 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5124
W20260620 18:49:48.509727 5109 connection.cc:535] client connection to 127.15.102.254:43491 recv error: Network error: failed to read from TLS socket (remote: 127.15.102.254:43491): Connection reset by peer (error 104)
2026-06-20T18:49:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1896 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:48 dist-test-slave-zfp8 krb5kdc[5222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:48 dist-test-slave-zfp8 krb5kdc[5222](info): set up 2 sockets
Jun 20 18:49:48 dist-test-slave-zfp8 krb5kdc[5222](info): commencing operation
krb5kdc: starting...
W20260620 18:49:49.607743 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:49 dist-test-slave-zfp8 krb5kdc[5222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:49.859563 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:42901
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:42901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:50.185724 5235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:50.186029 5235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:50.186223 5235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:50.193413 5235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:50.193529 5235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:50.193660 5235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:50.193774 5235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:50.193862 5235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:50.203709 5235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:42901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:42901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:50.205093 5235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:50.206480 5235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:50.218433 5243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:50.218488 5240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:50.218441 5241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:50.219172 5235 server_base.cc:1034] running on GCE node
I20260620 18:49:50.219965 5235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:50.221284 5235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:50.222524 5235 hybrid_clock.cc:648] HybridClock initialized: now 1781981390222480 us; error 54 us; skew 500 ppm
I20260620 18:49:50.222884 5235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:50 dist-test-slave-zfp8 krb5kdc[5222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981390, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:50.229846 5235 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:50.231531 5235 webserver.cc:460] Webserver started at http://127.15.102.254:35081/ using document root <none> and password file <none>
I20260620 18:49:50.232247 5235 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:50.232443 5235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:50.232780 5235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:50.235941 5235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "9b2b5ea9d5da4b56a529d38017748455"
format_stamp: "Formatted at 2026-06-20 18:49:50 on dist-test-slave-zfp8"
server_key: "b7f4c2be305e0753e2415a53cc4a55fb"
server_key_iv: "51c931b0e210cfdca99b392eb80b3d24"
server_key_version: "encryptionkey@0"
I20260620 18:49:50.236716 5235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "9b2b5ea9d5da4b56a529d38017748455"
format_stamp: "Formatted at 2026-06-20 18:49:50 on dist-test-slave-zfp8"
server_key: "b7f4c2be305e0753e2415a53cc4a55fb"
server_key_iv: "51c931b0e210cfdca99b392eb80b3d24"
server_key_version: "encryptionkey@0"
I20260620 18:49:50.242211 5235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 18:49:50.245750 5250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:50.247323 5235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 18:49:50.247517 5235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "9b2b5ea9d5da4b56a529d38017748455"
format_stamp: "Formatted at 2026-06-20 18:49:50 on dist-test-slave-zfp8"
server_key: "b7f4c2be305e0753e2415a53cc4a55fb"
server_key_iv: "51c931b0e210cfdca99b392eb80b3d24"
server_key_version: "encryptionkey@0"
I20260620 18:49:50.247742 5235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:50.253403 5235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:50.257364 5235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:50.257675 5235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:50.276603 5235 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:42901
I20260620 18:49:50.276654 5301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:42901 every 8 connection(s)
I20260620 18:49:50.278080 5235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:50.281903 5302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:50.283810 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5235
I20260620 18:49:50.283995 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:50.284554 15771 external_mini_cluster.cc:1428] Setting key 9ddee8941a742d79c86b7079e6607fd1
I20260620 18:49:50.292268 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:50.286384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47757 (local address 127.15.102.254:42901)
0620 18:49:50.287299 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:50.287314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:50.287370 (+ 56us) server_negotiation.cc:407] Connection header received
0620 18:49:50.288928 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:50.288949 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:50.289311 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:50.289654 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:50.290681 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:50.290687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:50.290699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:50.290713 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:50.290772 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:50.290791 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:50.290953 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:50.291084 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 18:49:50.291304 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":391,"threads_started":1}
I20260620 18:49:50.296118 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:50.296329 5302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455: Bootstrap starting.
I20260620 18:49:50.300181 5302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:50.301481 5302 log.cc:824] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:50.303768 5305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:50.301274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53057 (local address 127.15.102.254:42901)
0620 18:49:50.301462 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:50.301474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:50.301854 (+ 380us) server_negotiation.cc:407] Connection header received
0620 18:49:50.301976 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:50.301984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:50.302067 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:50.302195 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:50.303161 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:50.303166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:50.303171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:50.303188 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:50.303229 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:50.303282 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:50.303400 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:50.303454 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 18:49:50.303505 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 18:49:50.304924 5302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455: No bootstrap required, opened a new log
I20260620 18:49:50.312297 5302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b5ea9d5da4b56a529d38017748455" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42901 } }
I20260620 18:49:50.312681 5302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:50.312796 5302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b2b5ea9d5da4b56a529d38017748455, State: Initialized, Role: FOLLOWER
I20260620 18:49:50.313357 5302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b5ea9d5da4b56a529d38017748455" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42901 } }
I20260620 18:49:50.313594 5302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:50.313748 5302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:50.313915 5302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:50.316227 5302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b5ea9d5da4b56a529d38017748455" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42901 } }
I20260620 18:49:50.316728 5302 leader_election.cc:302] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b2b5ea9d5da4b56a529d38017748455; no voters:
I20260620 18:49:50.317281 5302 leader_election.cc:288] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:50.317453 5312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:50.318104 5312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [term 1 LEADER]: Becoming Leader. State: Replica: 9b2b5ea9d5da4b56a529d38017748455, State: Running, Role: LEADER
I20260620 18:49:50.318598 5312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b5ea9d5da4b56a529d38017748455" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42901 } }
I20260620 18:49:50.319382 5302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:50.321774 5314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b2b5ea9d5da4b56a529d38017748455. Latest consensus state: current_term: 1 leader_uuid: "9b2b5ea9d5da4b56a529d38017748455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b5ea9d5da4b56a529d38017748455" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42901 } } }
I20260620 18:49:50.322270 5314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:50.322108 5313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b2b5ea9d5da4b56a529d38017748455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b2b5ea9d5da4b56a529d38017748455" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 42901 } } }
I20260620 18:49:50.322556 5313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:50.325001 5320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:50.330022 5320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:50.338925 5320 catalog_manager.cc:1347] Generated new cluster ID: 089ad95f70f54333a3ad8b1e139eb39c
I20260620 18:49:50.339069 5320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:50.386456 5320 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:50.386998 5320 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:50.399811 5320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b2b5ea9d5da4b56a529d38017748455: Generated new TSK 0
I20260620 18:49:50.400844 5320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:50.421185 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5235
W20260620 18:49:50.422396 5297 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47757
W20260620 18:49:50.423162 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5235: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:50.423352 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5235
2026-06-20T18:49:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:50 dist-test-slave-zfp8 krb5kdc[5333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:50 dist-test-slave-zfp8 krb5kdc[5333](info): set up 2 sockets
Jun 20 18:49:50 dist-test-slave-zfp8 krb5kdc[5333](info): commencing operation
krb5kdc: starting...
W20260620 18:49:51.539690 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:51 dist-test-slave-zfp8 krb5kdc[5333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:51.794759 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:33017
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:33017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:52.120518 5346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:52.120811 5346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:52.120986 5346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:52.128190 5346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:52.128331 5346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:52.128434 5346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:52.128535 5346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:52.128608 5346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:52.138429 5346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:33017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:33017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:52.139810 5346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:52.141153 5346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:52.152952 5351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:52.152999 5352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:52.153144 5354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:52.154145 5346 server_base.cc:1034] running on GCE node
I20260620 18:49:52.154886 5346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:52.156379 5346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:52.157586 5346 hybrid_clock.cc:648] HybridClock initialized: now 1781981392157550 us; error 35 us; skew 500 ppm
I20260620 18:49:52.157996 5346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:52 dist-test-slave-zfp8 krb5kdc[5333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:52.161060 5346 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:52.162417 5346 webserver.cc:460] Webserver started at http://127.15.102.254:42429/ using document root <none> and password file <none>
I20260620 18:49:52.163103 5346 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:52.163204 5346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:52.163530 5346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:52.166534 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "b44c3cb50d5f42169b565a03a6487ecb"
format_stamp: "Formatted at 2026-06-20 18:49:52 on dist-test-slave-zfp8"
server_key: "9e9c018963c0bcb8670d2ce405e8b6c4"
server_key_iv: "da423a77144ac0a7e0889877903707b4"
server_key_version: "encryptionkey@0"
I20260620 18:49:52.167315 5346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "b44c3cb50d5f42169b565a03a6487ecb"
format_stamp: "Formatted at 2026-06-20 18:49:52 on dist-test-slave-zfp8"
server_key: "9e9c018963c0bcb8670d2ce405e8b6c4"
server_key_iv: "da423a77144ac0a7e0889877903707b4"
server_key_version: "encryptionkey@0"
I20260620 18:49:52.175798 5346 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260620 18:49:52.181293 5361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:52.182816 5346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260620 18:49:52.183019 5346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "b44c3cb50d5f42169b565a03a6487ecb"
format_stamp: "Formatted at 2026-06-20 18:49:52 on dist-test-slave-zfp8"
server_key: "9e9c018963c0bcb8670d2ce405e8b6c4"
server_key_iv: "da423a77144ac0a7e0889877903707b4"
server_key_version: "encryptionkey@0"
I20260620 18:49:52.183228 5346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:52.199797 5346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:52.206235 5346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:52.206586 5346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:52.227356 5346 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:33017
I20260620 18:49:52.227362 5412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:33017 every 8 connection(s)
I20260620 18:49:52.228802 5346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:52.229501 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5346
I20260620 18:49:52.229707 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:52.230225 15771 external_mini_cluster.cc:1428] Setting key b4b62ba349ea96924d2706ce2fc29cee
I20260620 18:49:52.233471 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 18:49:52 dist-test-slave-zfp8 krb5kdc[5333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:52.246227 5413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb: Bootstrap starting.
I20260620 18:49:52.250497 5413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:52.251835 5413 log.cc:824] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:52.255324 5413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb: No bootstrap required, opened a new log
I20260620 18:49:52.255347 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:52.232180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40001 (local address 127.15.102.254:33017)
0620 18:49:52.232956 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:52.232974 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:52.233028 (+ 54us) server_negotiation.cc:407] Connection header received
0620 18:49:52.235373 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:52.235402 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:52.235886 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:52.236346 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:52.237626 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:52.239764 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:52.241010 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:52.242019 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:52.242202 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:52.246643 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:52.246653 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:52.246674 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:52.246721 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:52.250746 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:52.251467 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:52.251482 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:52.251494 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:52.251709 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:52.252095 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:52.252100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:52.252104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:52.252754 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:52.253013 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:52.253538 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 18:49:52.253959 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":324,"threads_started":1}
I20260620 18:49:52.259167 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:52.264055 5413 raft_consensus.cc:348] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c3cb50d5f42169b565a03a6487ecb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 33017 } }
I20260620 18:49:52.264483 5413 raft_consensus.cc:374] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:52.264617 5413 raft_consensus.cc:729] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44c3cb50d5f42169b565a03a6487ecb, State: Initialized, Role: FOLLOWER
I20260620 18:49:52.265165 5413 consensus_queue.cc:260] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c3cb50d5f42169b565a03a6487ecb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 33017 } }
I20260620 18:49:52.265404 5413 raft_consensus.cc:388] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:52.265556 5413 raft_consensus.cc:482] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:52.265730 5413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:52.268075 5413 raft_consensus.cc:504] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c3cb50d5f42169b565a03a6487ecb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 33017 } }
I20260620 18:49:52.268563 5413 leader_election.cc:302] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b44c3cb50d5f42169b565a03a6487ecb; no voters:
I20260620 18:49:52.269273 5413 leader_election.cc:288] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:52.269449 5423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:52.271514 5423 raft_consensus.cc:686] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [term 1 LEADER]: Becoming Leader. State: Replica: b44c3cb50d5f42169b565a03a6487ecb, State: Running, Role: LEADER
I20260620 18:49:52.272609 5423 consensus_queue.cc:237] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c3cb50d5f42169b565a03a6487ecb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 33017 } }
I20260620 18:49:52.273793 5413 sys_catalog.cc:564] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:52.276291 5424 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b44c3cb50d5f42169b565a03a6487ecb. Latest consensus state: current_term: 1 leader_uuid: "b44c3cb50d5f42169b565a03a6487ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c3cb50d5f42169b565a03a6487ecb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 33017 } } }
I20260620 18:49:52.276786 5424 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:52.276751 5425 sys_catalog.cc:455] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b44c3cb50d5f42169b565a03a6487ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b44c3cb50d5f42169b565a03a6487ecb" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 33017 } } }
I20260620 18:49:52.277328 5425 sys_catalog.cc:458] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:52.279942 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:52.263554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35989 (local address 127.15.102.254:33017)
0620 18:49:52.263722 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:52.263726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:52.264260 (+ 534us) server_negotiation.cc:407] Connection header received
0620 18:49:52.264487 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:52.264498 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:52.264563 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:52.264687 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:52.266455 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:52.267385 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:52.268789 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:52.269480 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:52.269637 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:52.272560 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:52.272569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:52.272577 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:52.272628 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:52.276721 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:52.277682 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:52.277690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:52.277694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:52.277861 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:52.278263 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:52.278270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:52.278274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:52.278588 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:52.278889 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:52.279282 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 18:49:52.279506 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 18:49:52.291277 5439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:49:52.291430 5439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:49:52.291662 5428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:52.294044 5428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:52.303753 5428 catalog_manager.cc:1347] Generated new cluster ID: a6da8c71daf548868ee34d7621a1b127
I20260620 18:49:52.303942 5428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:52.324141 5428 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:52.325982 5428 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:52.336076 5428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b44c3cb50d5f42169b565a03a6487ecb: Generated new TSK 0
I20260620 18:49:52.337107 5428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:52.343886 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5346
I20260620 18:49:52.345238 5408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:49:52.434957 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5346
2026-06-20T18:49:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:52 dist-test-slave-zfp8 krb5kdc[5445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:52 dist-test-slave-zfp8 krb5kdc[5445](info): set up 2 sockets
Jun 20 18:49:52 dist-test-slave-zfp8 krb5kdc[5445](info): commencing operation
krb5kdc: starting...
W20260620 18:49:53.539711 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.011s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:53 dist-test-slave-zfp8 krb5kdc[5445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:53.785944 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:45435
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:54.114818 5458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:54.115108 5458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:54.115202 5458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:54.122452 5458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:54.122578 5458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:54.122627 5458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:54.122687 5458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:54.122738 5458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:54.132951 5458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:54.134256 5458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:54.135596 5458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:54.147987 5463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:54.148034 5466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:54.148034 5464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:54.148569 5458 server_base.cc:1034] running on GCE node
I20260620 18:49:54.149343 5458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:54.150772 5458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:54.152000 5458 hybrid_clock.cc:648] HybridClock initialized: now 1781981394151964 us; error 52 us; skew 500 ppm
I20260620 18:49:54.152343 5458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:54 dist-test-slave-zfp8 krb5kdc[5445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981394, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:54.155551 5458 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:54.156975 5458 webserver.cc:460] Webserver started at http://127.15.102.254:33207/ using document root <none> and password file <none>
I20260620 18:49:54.157667 5458 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:54.157814 5458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:54.158118 5458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:54.161283 5458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "49cdfe5d4a254bafb35a249b2fea0c01"
format_stamp: "Formatted at 2026-06-20 18:49:54 on dist-test-slave-zfp8"
server_key: "446a83ebe81a8797d3043e0f3e26fce7"
server_key_iv: "6d84a60184061e820908cafaaeb2ca3b"
server_key_version: "encryptionkey@0"
I20260620 18:49:54.162068 5458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "49cdfe5d4a254bafb35a249b2fea0c01"
format_stamp: "Formatted at 2026-06-20 18:49:54 on dist-test-slave-zfp8"
server_key: "446a83ebe81a8797d3043e0f3e26fce7"
server_key_iv: "6d84a60184061e820908cafaaeb2ca3b"
server_key_version: "encryptionkey@0"
I20260620 18:49:54.167387 5458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 18:49:54.170845 5473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:54.172194 5458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 18:49:54.172371 5458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "49cdfe5d4a254bafb35a249b2fea0c01"
format_stamp: "Formatted at 2026-06-20 18:49:54 on dist-test-slave-zfp8"
server_key: "446a83ebe81a8797d3043e0f3e26fce7"
server_key_iv: "6d84a60184061e820908cafaaeb2ca3b"
server_key_version: "encryptionkey@0"
I20260620 18:49:54.172575 5458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:54.191946 5458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:54.195869 5458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:54.196154 5458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:54.214663 5458 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:45435
I20260620 18:49:54.214682 5524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:45435 every 8 connection(s)
I20260620 18:49:54.216097 5458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:54.219800 5525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:54.221340 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5458
I20260620 18:49:54.221547 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:54.222106 15771 external_mini_cluster.cc:1428] Setting key 6e40a9c1c230adbdf92e1425140cd6cd
I20260620 18:49:54.233994 5525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01: Bootstrap starting.
I20260620 18:49:54.235307 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:54.223982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43715 (local address 127.15.102.254:45435)
0620 18:49:54.224785 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:54.224799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:54.224842 (+ 43us) server_negotiation.cc:407] Connection header received
0620 18:49:54.226394 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:54.226417 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:54.226822 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:54.227146 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:54.227764 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:54.228998 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:54.230875 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:54.231684 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:54.231815 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:54.233079 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:54.233086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:54.233099 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:54.233639 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:54.233715 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:54.233738 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:54.233825 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:54.234056 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 18:49:54.234314 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":392,"threads_started":1}
I20260620 18:49:54.237323 5525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:54.238521 5525 log.cc:824] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:54.238871 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:54.241707 5525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01: No bootstrap required, opened a new log
I20260620 18:49:54.250605 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:54.243858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57855 (local address 127.15.102.254:45435)
0620 18:49:54.244044 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:54.244051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:54.244581 (+ 530us) server_negotiation.cc:407] Connection header received
0620 18:49:54.244731 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:54.244736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:54.244826 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:54.244944 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:54.245377 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:54.246549 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:54.247947 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:54.248657 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:54.248801 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:54.249825 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:54.249834 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:54.249841 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:54.249869 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:54.249917 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:54.249936 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:54.250029 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:54.250149 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 18:49:54.250254 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:49:54.251626 5525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cdfe5d4a254bafb35a249b2fea0c01" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45435 } }
I20260620 18:49:54.252055 5525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:54.252164 5525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49cdfe5d4a254bafb35a249b2fea0c01, State: Initialized, Role: FOLLOWER
I20260620 18:49:54.252797 5525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cdfe5d4a254bafb35a249b2fea0c01" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45435 } }
I20260620 18:49:54.253057 5525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:54.253201 5525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:54.253379 5525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:54.255649 5525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cdfe5d4a254bafb35a249b2fea0c01" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45435 } }
I20260620 18:49:54.256168 5525 leader_election.cc:302] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49cdfe5d4a254bafb35a249b2fea0c01; no voters:
I20260620 18:49:54.256733 5525 leader_election.cc:288] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:54.256881 5535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:54.258059 5535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [term 1 LEADER]: Becoming Leader. State: Replica: 49cdfe5d4a254bafb35a249b2fea0c01, State: Running, Role: LEADER
I20260620 18:49:54.258661 5535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cdfe5d4a254bafb35a249b2fea0c01" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45435 } }
I20260620 18:49:54.259045 5525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:54.261754 5537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49cdfe5d4a254bafb35a249b2fea0c01. Latest consensus state: current_term: 1 leader_uuid: "49cdfe5d4a254bafb35a249b2fea0c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cdfe5d4a254bafb35a249b2fea0c01" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45435 } } }
I20260620 18:49:54.262115 5537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:54.261700 5536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49cdfe5d4a254bafb35a249b2fea0c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cdfe5d4a254bafb35a249b2fea0c01" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 45435 } } }
I20260620 18:49:54.262651 5536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01 [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:54.263222 5542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:54.269429 5542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:54.278529 5542 catalog_manager.cc:1347] Generated new cluster ID: 6ea7debc5ebf45818942ce5150c2c7b2
I20260620 18:49:54.278648 5542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:54.300838 5542 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:54.302762 5542 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:54.314795 5542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49cdfe5d4a254bafb35a249b2fea0c01: Generated new TSK 0
I20260620 18:49:54.315828 5542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:54.320753 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5458
W20260620 18:49:54.322425 5520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43715
W20260620 18:49:54.323074 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5458: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:54.323206 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5458
2026-06-20T18:49:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:54 dist-test-slave-zfp8 krb5kdc[5556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:54 dist-test-slave-zfp8 krb5kdc[5556](info): set up 2 sockets
Jun 20 18:49:54 dist-test-slave-zfp8 krb5kdc[5556](info): commencing operation
krb5kdc: starting...
W20260620 18:49:55.439788 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:55 dist-test-slave-zfp8 krb5kdc[5556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:55.692193 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37791
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:37791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:56.012651 5569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:56.012924 5569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:56.013105 5569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:56.020174 5569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:56.020313 5569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:56.020423 5569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:56.020536 5569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:56.020617 5569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:56.030198 5569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:37791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:37791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:56.031555 5569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:56.032882 5569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:56.044425 5574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:56.044433 5575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:56.044528 5577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:56.045202 5569 server_base.cc:1034] running on GCE node
I20260620 18:49:56.045960 5569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:56.047377 5569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:56.048573 5569 hybrid_clock.cc:648] HybridClock initialized: now 1781981396048538 us; error 29 us; skew 500 ppm
I20260620 18:49:56.048913 5569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:56 dist-test-slave-zfp8 krb5kdc[5556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981396, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:56.056689 5569 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:56.058223 5569 webserver.cc:460] Webserver started at http://127.15.102.254:42177/ using document root <none> and password file <none>
I20260620 18:49:56.058898 5569 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:56.059042 5569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:56.059425 5569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:56.062376 5569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "c2690e10832c4a48ba29ca385f59c49a"
format_stamp: "Formatted at 2026-06-20 18:49:56 on dist-test-slave-zfp8"
server_key: "c39223ac0fbd11481833a7f4b56a5c8b"
server_key_iv: "ba4d96017a1130af4a09fa0ffb7b0db6"
server_key_version: "encryptionkey@0"
I20260620 18:49:56.063133 5569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "c2690e10832c4a48ba29ca385f59c49a"
format_stamp: "Formatted at 2026-06-20 18:49:56 on dist-test-slave-zfp8"
server_key: "c39223ac0fbd11481833a7f4b56a5c8b"
server_key_iv: "ba4d96017a1130af4a09fa0ffb7b0db6"
server_key_version: "encryptionkey@0"
I20260620 18:49:56.068334 5569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260620 18:49:56.071918 5584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:56.073252 5569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 18:49:56.073429 5569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "c2690e10832c4a48ba29ca385f59c49a"
format_stamp: "Formatted at 2026-06-20 18:49:56 on dist-test-slave-zfp8"
server_key: "c39223ac0fbd11481833a7f4b56a5c8b"
server_key_iv: "ba4d96017a1130af4a09fa0ffb7b0db6"
server_key_version: "encryptionkey@0"
I20260620 18:49:56.073625 5569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:56.079511 5569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:56.083281 5569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:56.083568 5569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:56.101606 5569 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:37791
I20260620 18:49:56.101636 5635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:37791 every 8 connection(s)
I20260620 18:49:56.103012 5569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:56.106436 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5569
I20260620 18:49:56.106621 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:56.106877 5636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:56.107183 15771 external_mini_cluster.cc:1428] Setting key e9b8098625973b6232198dde9f4076a1
I20260620 18:49:56.114746 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:56.108892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59673 (local address 127.15.102.254:37791)
0620 18:49:56.109682 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:56.109693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:56.109741 (+ 48us) server_negotiation.cc:407] Connection header received
0620 18:49:56.111578 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:56.111599 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:56.111929 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:56.112257 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:56.113281 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:56.113287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:56.113298 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:56.113311 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:56.113368 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:56.113390 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:56.113557 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:56.113663 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 18:49:56.113863 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":346,"threads_started":1}
I20260620 18:49:56.118523 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:56.120653 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a: Bootstrap starting.
I20260620 18:49:56.124680 5636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:56.125901 5636 log.cc:824] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:56.126148 5639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:56.123706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54201 (local address 127.15.102.254:37791)
0620 18:49:56.123903 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:56.123908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:56.124279 (+ 371us) server_negotiation.cc:407] Connection header received
0620 18:49:56.124400 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:56.124405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:56.124469 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:56.124592 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:56.125489 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:56.125494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:56.125498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 18:49:56.125515 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:56.125572 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 18:49:56.125592 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:56.125716 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:56.125787 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 18:49:56.125841 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 18:49:56.128705 5636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a: No bootstrap required, opened a new log
I20260620 18:49:56.136382 5636 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2690e10832c4a48ba29ca385f59c49a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37791 } }
I20260620 18:49:56.136770 5636 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 18:49:56.136874 5636 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2690e10832c4a48ba29ca385f59c49a, State: Initialized, Role: FOLLOWER
I20260620 18:49:56.137558 5636 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2690e10832c4a48ba29ca385f59c49a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37791 } }
I20260620 18:49:56.137825 5636 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 18:49:56.137991 5636 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 18:49:56.138193 5636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:56.140427 5636 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2690e10832c4a48ba29ca385f59c49a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37791 } }
I20260620 18:49:56.140904 5636 leader_election.cc:302] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2690e10832c4a48ba29ca385f59c49a; no voters:
I20260620 18:49:56.141489 5636 leader_election.cc:288] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 18:49:56.141616 5646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:56.142804 5646 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [term 1 LEADER]: Becoming Leader. State: Replica: c2690e10832c4a48ba29ca385f59c49a, State: Running, Role: LEADER
I20260620 18:49:56.143453 5646 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2690e10832c4a48ba29ca385f59c49a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37791 } }
I20260620 18:49:56.143626 5636 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:56.146066 5648 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2690e10832c4a48ba29ca385f59c49a. Latest consensus state: current_term: 1 leader_uuid: "c2690e10832c4a48ba29ca385f59c49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2690e10832c4a48ba29ca385f59c49a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37791 } } }
I20260620 18:49:56.146474 5648 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:56.147013 5647 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2690e10832c4a48ba29ca385f59c49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2690e10832c4a48ba29ca385f59c49a" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 37791 } } }
I20260620 18:49:56.147395 5647 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:56.148006 5655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:56.158646 5655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:56.167626 5655 catalog_manager.cc:1347] Generated new cluster ID: a3d5ef8b132949479a37f4ce9a4451b4
I20260620 18:49:56.167755 5655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:56.195922 5655 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:56.196431 5655 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:56.204968 5655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2690e10832c4a48ba29ca385f59c49a: Generated new TSK 0
I20260620 18:49:56.205988 5655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:56.217839 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5569
W20260620 18:49:56.218935 5631 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59673
W20260620 18:49:56.219643 15771 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 18:49:56.219784 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5569
2026-06-20T18:49:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1897 ms)
[----------] 18 tests from AuthTokenIssuingTest (35141 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:49:56 dist-test-slave-zfp8 krb5kdc[5668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:49:56 dist-test-slave-zfp8 krb5kdc[5668](info): set up 2 sockets
Jun 20 18:49:56 dist-test-slave-zfp8 krb5kdc[5668](info): commencing operation
krb5kdc: starting...
W20260620 18:49:57.331915 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:49:57 dist-test-slave-zfp8 krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:49:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:49:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:49:57.586477 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:39339
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:39339,127.15.102.253:45841,127.15.102.252:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:49:57.912559 5681 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:49:57.912943 5681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:57.913069 5681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:57.913203 5681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:57.920701 5681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:57.920837 5681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:57.920974 5681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:57.921056 5681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:57.921147 5681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:57.930966 5681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:39339,127.15.102.253:45841,127.15.102.252:39489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:39339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:57.932331 5681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:57.933629 5681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:57.945577 5687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:57.945617 5689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:57.945670 5686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:57.946381 5681 server_base.cc:1034] running on GCE node
I20260620 18:49:57.947053 5681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:57.948421 5681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:57.949636 5681 hybrid_clock.cc:648] HybridClock initialized: now 1781981397949557 us; error 47 us; skew 500 ppm
I20260620 18:49:57.949985 5681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:57 dist-test-slave-zfp8 krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981397, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:57.953076 5681 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:49:57.954412 5681 webserver.cc:460] Webserver started at http://127.15.102.254:41455/ using document root <none> and password file <none>
I20260620 18:49:57.955190 5681 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:57.955333 5681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:57.955690 5681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:57.958616 5681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "76e659a26c6d4f44b015475e20d0befd"
format_stamp: "Formatted at 2026-06-20 18:49:57 on dist-test-slave-zfp8"
server_key: "ca091951ca9b1b03c1bc8eb3d6ab5a1e"
server_key_iv: "feb5ec77d98c6f3b5a3a84c037852829"
server_key_version: "encryptionkey@0"
I20260620 18:49:57.959400 5681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "76e659a26c6d4f44b015475e20d0befd"
format_stamp: "Formatted at 2026-06-20 18:49:57 on dist-test-slave-zfp8"
server_key: "ca091951ca9b1b03c1bc8eb3d6ab5a1e"
server_key_iv: "feb5ec77d98c6f3b5a3a84c037852829"
server_key_version: "encryptionkey@0"
I20260620 18:49:57.964391 5681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 18:49:57.967749 5696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:57.969027 5681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 18:49:57.969216 5681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "76e659a26c6d4f44b015475e20d0befd"
format_stamp: "Formatted at 2026-06-20 18:49:57 on dist-test-slave-zfp8"
server_key: "ca091951ca9b1b03c1bc8eb3d6ab5a1e"
server_key_iv: "feb5ec77d98c6f3b5a3a84c037852829"
server_key_version: "encryptionkey@0"
I20260620 18:49:57.969434 5681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:58.000521 5681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:58.005079 5681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:58.005381 5681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:58.023950 5681 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:39339
I20260620 18:49:58.023988 5747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:39339 every 8 connection(s)
I20260620 18:49:58.025345 5681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:49:58.029047 5748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:58.031462 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5681
I20260620 18:49:58.031675 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:49:58.032389 15771 external_mini_cluster.cc:1428] Setting key e023337be0b13129eb96a499fc817034
I20260620 18:49:58.035753 5748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:49:58 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981397, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.102.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.253@KRBTEST.COM" created.
I20260620 18:49:58.062064 5750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.038864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.15.102.254:39339)
0620 18:49:58.039771 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:58.039787 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:58.040339 (+ 552us) server_negotiation.cc:407] Connection header received
0620 18:49:58.043393 (+ 3054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:58.043400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:58.043693 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:58.043872 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:58.045496 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.046774 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.049039 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.049624 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.049777 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:58.053983 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:58.053992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:58.054005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:58.054035 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:58.057582 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:58.058365 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:58.058413 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:58.058415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:58.058580 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:58.059127 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:58.059135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:58.059138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:58.059696 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:58.059866 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:58.060586 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 18:49:58.061001 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":643,"thread_start_us":396,"threads_started":1}
I20260620 18:49:58.062120 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.038864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:39339 (local address 127.0.0.1:38642)
0620 18:49:58.040164 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.040205 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:58.042924 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:58.043951 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:58.043965 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:58.044799 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:58.045276 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:58.045302 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.046946 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.046954 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:58.048880 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:58.048897 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.049867 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.049877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:58.050099 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:58.050845 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:58.050878 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:58.053732 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:58.057718 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:58.057731 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:58.057744 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:58.058190 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:58.058700 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:58.058708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:58.058714 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:58.058945 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:58.059857 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:58.059868 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:58.060166 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 18:49:58.060488 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 18:49:58.061200 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":146,"spinlock_wait_cycles":50944,"thread_start_us":815,"threads_started":1}
I20260620 18:49:58.065056 5748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:49:58.066257 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.065881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53180)
0620 18:49:58.066041 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.066140 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:49:58.066735 5699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
I20260620 18:49:58.068383 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.068001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53192)
0620 18:49:58.068204 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.068270 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 18:49:58.068710 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:45841: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab.
Entry for principal kudu/127.15.102.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab.
I20260620 18:49:58.090225 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.253:45841
--webserver_interface=127.15.102.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab
--principal=kudu/127.15.102.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:39339,127.15.102.253:45841,127.15.102.252:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:49:58.118072 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } attempt: 1
I20260620 18:49:58.119441 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.118895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53198)
0620 18:49:58.119105 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.119258 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 18:49:58.120543 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.120238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53210)
0620 18:49:58.120367 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.120429 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 18:49:58.120819 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:45841: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
I20260620 18:49:58.189123 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } attempt: 2
I20260620 18:49:58.190491 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.189957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53220)
0620 18:49:58.190238 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.190352 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 18:49:58.191744 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.191372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53224)
0620 18:49:58.191533 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.191612 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:49:58.192015 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:45841: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
I20260620 18:49:58.283335 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } attempt: 3
I20260620 18:49:58.284627 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.284184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53238)
0620 18:49:58.284399 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.284492 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 18:49:58.285821 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.285412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53252)
0620 18:49:58.285610 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.285681 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 18:49:58.286105 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:45841: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
W20260620 18:49:58.428901 5755 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:49:58.429251 5755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:58.429330 5755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:58.429491 5755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 18:49:58.429450 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } attempt: 4
I20260620 18:49:58.430757 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.430192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53254)
0620 18:49:58.430517 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.430624 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 18:49:58.431792 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.431451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53260)
0620 18:49:58.431597 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.431673 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 18:49:58.432085 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:45841: Network error: Client connection negotiation failed: client connection to 127.15.102.253:45841: connect: Connection refused (error 111)
W20260620 18:49:58.437521 5755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:58.437608 5755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:58.437723 5755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:58.437808 5755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:58.437922 5755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:58.448124 5755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:39339,127.15.102.253:45841,127.15.102.252:39489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab
--principal=kudu/127.15.102.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.253:45841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.102.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:58.449450 5755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:58.450906 5755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:49:58.465016 5760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:58.465145 5763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:58.465050 5761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:58.466497 5755 server_base.cc:1034] running on GCE node
I20260620 18:49:58.467355 5755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:58.468729 5755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:58.469982 5755 hybrid_clock.cc:648] HybridClock initialized: now 1781981398469918 us; error 36 us; skew 500 ppm
I20260620 18:49:58.470337 5755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:58 dist-test-slave-zfp8 krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:58.473853 5755 init.cc:377] Logged in from keytab as kudu/127.15.102.253@KRBTEST.COM (short username kudu)
I20260620 18:49:58.475302 5755 webserver.cc:460] Webserver started at http://127.15.102.253:38345/ using document root <none> and password file <none>
I20260620 18:49:58.475957 5755 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:58.476046 5755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:58.476315 5755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:58.479126 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/data/instance:
uuid: "0b6f12b01d114e878c15d04ccd98426f"
format_stamp: "Formatted at 2026-06-20 18:49:58 on dist-test-slave-zfp8"
server_key: "2fd829a6d2fdecb350ed0a9adb88746a"
server_key_iv: "ccad16e4251d8624d1e54b4d27269cc4"
server_key_version: "encryptionkey@0"
I20260620 18:49:58.479923 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/wal/instance:
uuid: "0b6f12b01d114e878c15d04ccd98426f"
format_stamp: "Formatted at 2026-06-20 18:49:58 on dist-test-slave-zfp8"
server_key: "2fd829a6d2fdecb350ed0a9adb88746a"
server_key_iv: "ccad16e4251d8624d1e54b4d27269cc4"
server_key_version: "encryptionkey@0"
I20260620 18:49:58.484879 5755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 18:49:58.488149 5770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:58.489428 5755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 18:49:58.489612 5755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/wal
uuid: "0b6f12b01d114e878c15d04ccd98426f"
format_stamp: "Formatted at 2026-06-20 18:49:58 on dist-test-slave-zfp8"
server_key: "2fd829a6d2fdecb350ed0a9adb88746a"
server_key_iv: "ccad16e4251d8624d1e54b4d27269cc4"
server_key_version: "encryptionkey@0"
I20260620 18:49:58.489846 5755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:58.524143 5755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:58.528659 5755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:58.528954 5755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:58.547214 5755 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.253:45841
I20260620 18:49:58.547277 5821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.253:45841 every 8 connection(s)
I20260620 18:49:58.548615 5755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
I20260620 18:49:58.552435 5822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:58.556258 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5755
I20260620 18:49:58.556447 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-1/wal/instance
I20260620 18:49:58.557006 15771 external_mini_cluster.cc:1428] Setting key 05f2038cf8d7c6997ac720b0f1a25e40
I20260620 18:49:58.560993 5822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:49:58 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.102.252@KRBTEST.COM; defaulting to no policy
I20260620 18:49:58.584889 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.564487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.15.102.254:39339)
0620 18:49:58.565011 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:58.565020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:58.566214 (+ 1194us) server_negotiation.cc:407] Connection header received
0620 18:49:58.568591 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:58.568597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:58.568697 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:58.568820 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:58.570387 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.571108 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.572688 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.573172 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.573302 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:58.577162 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:58.577172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:58.577181 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:58.577252 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:58.581306 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:58.582116 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:58.582124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:58.582129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:58.582304 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:58.582726 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:58.582734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:58.582739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:58.583187 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:58.583412 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:58.584146 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 18:49:58.584391 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":255,"threads_started":1}
I20260620 18:49:58.585598 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.565126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:39339 (local address 127.0.0.1:38650)
0620 18:49:58.566004 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.566056 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:58.568286 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:58.569060 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:58.569068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:58.569585 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:58.570167 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:58.570185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.571296 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.571301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:58.572536 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:58.572548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.573405 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.573412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:58.573637 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:58.574405 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:58.574430 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:58.576904 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:58.581475 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:58.581483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:58.581498 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:58.581956 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:58.582425 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:58.582428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:58.582430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:58.582594 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:58.583357 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:58.583363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:58.583707 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 18:49:58.583949 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 18:49:58.584354 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.252@KRBTEST.COM" created.
I20260620 18:49:58.587519 5822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:49:58 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:49:58.604655 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.588425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53274)
0620 18:49:58.588658 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.588680 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:58.588792 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:58.589844 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:58.589851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:58.590329 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:58.590558 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:58.590572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.592070 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.592078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:58.593588 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:58.593604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.594675 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.594683 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:58.594917 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:58.595601 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:58.595629 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:58.597969 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:58.602326 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:58.602329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:58.602332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:58.602673 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:58.603080 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:58.603083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:58.603085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:58.603213 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:58.603981 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:58.603987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:58.604095 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 18:49:58.604203 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 18:49:58.604395 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 18:49:58.605161 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.588679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.15.102.253:45841)
0620 18:49:58.589406 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:58.589412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:58.589439 (+ 27us) server_negotiation.cc:407] Connection header received
0620 18:49:58.589483 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:58.589489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:58.589666 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:58.589792 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:58.590752 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.591918 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.593732 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.594447 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.594683 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:58.598187 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:58.598203 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:58.598223 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:58.598270 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:58.602090 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:58.602808 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:58.602819 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:58.602825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:58.602971 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:58.603370 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:58.603377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:58.603383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:58.603860 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:58.604082 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:58.604379 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 18:49:58.604657 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":425,"threads_started":1}
I20260620 18:49:58.607795 5822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:49:58.609036 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.608468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33776)
0620 18:49:58.608674 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.608843 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 18:49:58.609369 5771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
I20260620 18:49:58.612568 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.612120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33788)
0620 18:49:58.612325 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.612428 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:49:58.612777 5822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab.
Entry for principal kudu/127.15.102.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab.
I20260620 18:49:58.622470 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.252:39489
--webserver_interface=127.15.102.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab
--principal=kudu/127.15.102.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:39339,127.15.102.253:45841,127.15.102.252:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:49:58.662132 5822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 1
I20260620 18:49:58.663403 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.662821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33790)
0620 18:49:58.663042 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.663178 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 18:49:58.664503 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.664069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33806)
0620 18:49:58.664262 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.664361 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 18:49:58.664718 5822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
I20260620 18:49:58.731422 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } attempt: 5
I20260620 18:49:58.732946 5822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 2
I20260620 18:49:58.733932 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.733469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33810)
0620 18:49:58.733654 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.733751 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 18:49:58.735064 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.734657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33824)
0620 18:49:58.734821 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.734902 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 18:49:58.735417 5822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Jun 20 18:49:58 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981397, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:49:58.746055 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.732306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.15.102.253:45841)
0620 18:49:58.732463 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:58.732469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:58.732573 (+ 104us) server_negotiation.cc:407] Connection header received
0620 18:49:58.732734 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:58.732742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:58.732861 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:58.733001 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:58.733904 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.734945 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.736646 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.737076 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.737173 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:58.740543 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:58.740569 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:58.740576 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:58.740616 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:58.743452 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:58.744108 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:58.744115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:58.744119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:58.744242 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:58.744626 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:58.744632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:58.744635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:58.744922 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:58.745072 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:58.745457 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 18:49:58.745732 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 18:49:58.746073 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.732215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53290)
0620 18:49:58.732441 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.732463 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:58.732607 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:58.733055 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:58.733060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:58.733575 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:58.733744 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:58.733753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.735096 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.735101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:58.736489 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:58.736504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:58.737224 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:58.737229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:58.737431 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:58.738019 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:58.738041 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:58.740350 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:58.743587 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:58.743592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:58.743596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:58.743985 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:58.744369 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:58.744373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:58.744377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:58.744518 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:58.745052 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:58.745057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:58.745209 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 18:49:58.745363 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 18:49:58.745712 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 18:49:58.747419 5748 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:49:58.748461 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.748095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33836)
0620 18:49:58.748257 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.748334 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 18:49:58.749368 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.749043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33850)
0620 18:49:58.749186 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.749266 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:49:58.749588 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
I20260620 18:49:58.800817 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 1
I20260620 18:49:58.802213 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.801497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33858)
0620 18:49:58.801786 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.801962 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 18:49:58.803422 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.802946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33868)
0620 18:49:58.803151 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.803292 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 18:49:58.803668 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
I20260620 18:49:58.826678 5822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 3
I20260620 18:49:58.828078 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.827424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33884)
0620 18:49:58.827706 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.827842 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 18:49:58.829315 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.828849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33886)
0620 18:49:58.829075 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.829176 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260620 18:49:58.829558 5822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
I20260620 18:49:58.871903 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 2
I20260620 18:49:58.873178 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.872575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33898)
0620 18:49:58.872827 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.872953 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 18:49:58.874352 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.873951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33904)
0620 18:49:58.874100 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.874207 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 18:49:58.874598 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
W20260620 18:49:58.948393 5830 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:49:58.948748 5830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:49:58.948863 5830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:49:58.948989 5830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:49:58.956375 5830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:49:58.956468 5830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:49:58.956552 5830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:49:58.956641 5830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:49:58.956748 5830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:49:58.966770 5830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:39339,127.15.102.253:45841,127.15.102.252:39489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab
--principal=kudu/127.15.102.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.252:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.102.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:49:58.968134 5830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:49:58.969486 5830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 18:49:58.972805 5822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 4
I20260620 18:49:58.973941 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.973428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33908)
0620 18:49:58.973653 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.973743 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:49:58.975097 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.974639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33924)
0620 18:49:58.974824 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.974925 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 18:49:58.975376 5822 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
I20260620 18:49:58.980809 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 3
W20260620 18:49:58.981775 5836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:49:58.981860 5838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:58.982118 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.981444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33938)
0620 18:49:58.981735 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.981870 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260620 18:49:58.982273 5835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:49:58.983374 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:58.982967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33954)
0620 18:49:58.983155 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:58.983218 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 18:49:58.983592 5748 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:39489: Network error: Client connection negotiation failed: client connection to 127.15.102.252:39489: connect: Connection refused (error 111)
I20260620 18:49:58.984876 5830 server_base.cc:1034] running on GCE node
I20260620 18:49:58.985648 5830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:49:58.987079 5830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:49:58.988332 5830 hybrid_clock.cc:648] HybridClock initialized: now 1781981398988231 us; error 75 us; skew 500 ppm
I20260620 18:49:58.988687 5830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:49:58 dist-test-slave-zfp8 krb5kdc[5668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:49:58.991683 5830 init.cc:377] Logged in from keytab as kudu/127.15.102.252@KRBTEST.COM (short username kudu)
I20260620 18:49:58.993188 5830 webserver.cc:460] Webserver started at http://127.15.102.252:43491/ using document root <none> and password file <none>
I20260620 18:49:58.993949 5830 fs_manager.cc:359] Metadata directory not provided
I20260620 18:49:58.994074 5830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:49:58.994437 5830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:49:58.997372 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/data/instance:
uuid: "486e78b8caa6443abe1b1b9137ff655e"
format_stamp: "Formatted at 2026-06-20 18:49:58 on dist-test-slave-zfp8"
server_key: "baf4b0fdb05bdcf6ccef748f9c670a02"
server_key_iv: "51aa13c8d702b29d933889ebce7f1859"
server_key_version: "encryptionkey@0"
I20260620 18:49:58.998164 5830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/wal/instance:
uuid: "486e78b8caa6443abe1b1b9137ff655e"
format_stamp: "Formatted at 2026-06-20 18:49:58 on dist-test-slave-zfp8"
server_key: "baf4b0fdb05bdcf6ccef748f9c670a02"
server_key_iv: "51aa13c8d702b29d933889ebce7f1859"
server_key_version: "encryptionkey@0"
I20260620 18:49:59.003320 5830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:49:59.006628 5845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:49:59.008008 5830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:49:59.008204 5830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/wal
uuid: "486e78b8caa6443abe1b1b9137ff655e"
format_stamp: "Formatted at 2026-06-20 18:49:58 on dist-test-slave-zfp8"
server_key: "baf4b0fdb05bdcf6ccef748f9c670a02"
server_key_iv: "51aa13c8d702b29d933889ebce7f1859"
server_key_version: "encryptionkey@0"
I20260620 18:49:59.008421 5830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:49:59.021009 5830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:49:59.024953 5830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:49:59.025224 5830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:49:59.043520 5830 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.252:39489
I20260620 18:49:59.043529 5896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.252:39489 every 8 connection(s)
I20260620 18:49:59.044863 5830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
I20260620 18:49:59.046085 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 5830
I20260620 18:49:59.046267 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781981043007005-15771-0/minicluster-data/master-2/wal/instance
I20260620 18:49:59.046823 15771 external_mini_cluster.cc:1428] Setting key 90de9ad79a71f6dce6c55ea5b64d2028
I20260620 18:49:59.049438 5897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:49:59.058000 5897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:49:59 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
Jun 20 18:49:59 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:49:59.070129 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.048748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.15.102.252:39489)
0620 18:49:59.049523 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.049541 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.049591 (+ 50us) server_negotiation.cc:407] Connection header received
0620 18:49:59.051113 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.051136 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.051595 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.051938 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.053685 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.055019 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.056850 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.057914 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.058112 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.062480 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.062490 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.062502 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.062551 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.065710 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.066554 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.066564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.066575 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.066792 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.067176 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.067181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.067193 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.067871 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.068203 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.068474 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.068843 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":343,"threads_started":1}
I20260620 18:49:59.076267 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.060677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.15.102.254:39339)
0620 18:49:59.060879 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.060888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.061426 (+ 538us) server_negotiation.cc:407] Connection header received
0620 18:49:59.061690 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.061694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.061790 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.061904 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.063434 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.064121 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.065550 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.066126 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.066253 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.069988 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.069993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.069996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.070027 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.073274 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.074133 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.074137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.074139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.074328 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.074749 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.074756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.074761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.075142 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.075319 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.075807 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.076018 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 18:49:59.076411 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.060648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:39339 (local address 127.0.0.1:38660)
0620 18:49:59.061281 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:59.061327 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:59.061508 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:59.061962 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:59.061969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:59.062835 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:59.063278 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.063295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.064274 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.064279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.065379 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.065391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.066296 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.066300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.066421 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.067045 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:59.067065 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:59.069785 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:59.073401 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.073412 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.073419 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.074021 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.074455 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.074461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.074468 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.074618 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.075315 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:59.075322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:59.075473 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 18:49:59.075667 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 18:49:59.076017 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":316,"threads_started":1}
I20260620 18:49:59.077831 5897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:49:59 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
Jun 20 18:49:59 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:49:59.089095 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.073366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.15.102.254:39339)
0620 18:49:59.073966 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.073972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.073996 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:49:59.074044 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.074051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.074129 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.074235 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.075688 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.076975 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.078469 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.079086 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.079228 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.083611 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.083618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.083623 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.083662 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.086422 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.087205 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.087209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.087211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.087385 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.087816 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.087819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.087821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.088071 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.088277 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.088613 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.088771 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":245,"threads_started":1}
I20260620 18:49:59.092092 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.078585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:45841 (local address 127.0.0.1:53298)
0620 18:49:59.078765 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:59.078786 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:59.078882 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:59.079185 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:59.079189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:59.079572 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:59.079698 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.079706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.080635 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.080643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.081486 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.081496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.082486 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.082492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.082645 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.083174 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:59.083190 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:59.085548 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:59.089158 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.089163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.089167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.089660 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.090073 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.090076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.090079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.090280 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.091023 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:59.091030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:59.091205 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 18:49:59.091482 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 18:49:59.091777 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:49:59.092190 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.078603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.15.102.253:45841)
0620 18:49:59.078797 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.078803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.078833 (+ 30us) server_negotiation.cc:407] Connection header received
0620 18:49:59.078993 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.078997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.079055 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.079128 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.079846 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.080501 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.081629 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.082320 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.082454 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.085754 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.085760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.085763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.085796 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.089027 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.089797 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.089802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.089806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.089934 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.090389 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.090392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.090394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.090769 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.090933 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.091557 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.091790 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:49:59.093061 5897 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:49:59 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
Jun 20 18:49:59 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
I20260620 18:49:59.106542 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.093945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33972)
0620 18:49:59.094132 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:59.094146 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:59.094224 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:59.094495 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:59.094501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:59.094903 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:59.095028 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.095035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.096006 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.096013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.096942 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.096957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.097919 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.097924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.098090 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.098630 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:59.098646 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:59.100780 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:59.103811 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.103817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.103821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.104318 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.104734 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.104737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.104739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.104875 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.105454 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:59.105459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:59.105568 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 18:49:59.105744 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 18:49:59.106070 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":4}
I20260620 18:49:59.106642 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.093903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33972 (local address 127.15.102.252:39489)
0620 18:49:59.094071 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.094077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.094208 (+ 131us) server_negotiation.cc:407] Connection header received
0620 18:49:59.094318 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.094323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.094386 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.094470 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.095151 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.095888 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.097081 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.097756 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.097893 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.100987 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.100994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.101001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.101039 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.103683 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.104435 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.104438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.104441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.104616 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.104983 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.104989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.104994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.105320 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.105474 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.105827 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.106272 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":40}
I20260620 18:49:59.107350 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.090802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.15.102.253:45841)
0620 18:49:59.091422 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.091430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.091454 (+ 24us) server_negotiation.cc:407] Connection header received
0620 18:49:59.091503 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.091507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.091573 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.091667 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.093369 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.094563 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.095792 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.096208 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.096340 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.100805 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.100812 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.100818 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.100861 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.104432 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.105179 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.105183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.105185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.105310 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.105719 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.105723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.105726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.106069 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.106329 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.106648 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.106874 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":250,"threads_started":1}
I20260620 18:49:59.112208 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:49:59.112387 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: Bootstrap starting.
I20260620 18:49:59.116106 5897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:59.117417 5897 log.cc:824] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:59.114924 5723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "76e659a26c6d4f44b015475e20d0befd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38664
I20260620 18:49:59.120458 5897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: No bootstrap required, opened a new log
I20260620 18:49:59.131090 5897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } }
I20260620 18:49:59.131552 5897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486e78b8caa6443abe1b1b9137ff655e, State: Initialized, Role: FOLLOWER
I20260620 18:49:59.132043 5897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } }
I20260620 18:49:59.133090 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.133423 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:49:59.134266 5897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:49:59.143882 5916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:49:59.144006 5916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:49:59.141870 5830 master_runner.cc:186] Error getting master registration for 127.15.102.254:39339: OK, instance_id { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" instance_seqno: 1781981398018680 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:49:59.152477 5830 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:49:59.160775 5748 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 4
Jun 20 18:49:59 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981397, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
I20260620 18:49:59.175143 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.161513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:33986)
0620 18:49:59.161823 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:59.161841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:59.161940 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:59.162314 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:59.162319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:59.162718 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:59.162883 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.162892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.164028 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.164035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.164965 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.164976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.165754 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.165759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.165969 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.166619 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:59.166638 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:59.168962 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:59.172433 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.172438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.172442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.172879 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.173252 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.173258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.173261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.173430 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.174125 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:59.174131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:59.174288 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 18:49:59.174430 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 18:49:59.174726 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 18:49:59.176515 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.161694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.15.102.252:39489)
0620 18:49:59.161907 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.161914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.161937 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:49:59.162082 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.162094 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.162160 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.162279 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.163033 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.163892 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.165108 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.165597 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.165743 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.169163 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.169171 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.169178 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.169216 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.172250 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.172996 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.173002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.173006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.173141 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.173559 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.173565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.173568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.173978 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.174164 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.174541 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.174740 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:49:59.182328 5748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd: Bootstrap starting.
I20260620 18:49:59.186357 5748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:59.187649 5748 log.cc:824] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:59.190665 5748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd: No bootstrap required, opened a new log
I20260620 18:49:59.197523 5748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } }
I20260620 18:49:59.197921 5748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e659a26c6d4f44b015475e20d0befd, State: Initialized, Role: FOLLOWER
I20260620 18:49:59.198410 5748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } }
I20260620 18:49:59.199652 5919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.200254 5919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:49:59.200843 5748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:59.200920 5723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:49:59.201110 5723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:59.205315 5723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } }
I20260620 18:49:59.208211 5872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76e659a26c6d4f44b015475e20d0befd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "486e78b8caa6443abe1b1b9137ff655e"
I20260620 18:49:59.208822 5872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:59.209326 5723 leader_election.cc:288] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [CANDIDATE]: Term 1 election: Requested vote from peers 0b6f12b01d114e878c15d04ccd98426f (127.15.102.253:45841), 486e78b8caa6443abe1b1b9137ff655e (127.15.102.252:39489)
I20260620 18:49:59.212946 5872 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76e659a26c6d4f44b015475e20d0befd in term 1.
I20260620 18:49:59.213738 5697 leader_election.cc:302] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486e78b8caa6443abe1b1b9137ff655e, 76e659a26c6d4f44b015475e20d0befd; no voters:
I20260620 18:49:59.214324 5919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:49:59.221292 5919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [term 1 LEADER]: Becoming Leader. State: Replica: 76e659a26c6d4f44b015475e20d0befd, State: Running, Role: LEADER
I20260620 18:49:59.221988 5919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } }
I20260620 18:49:59.228773 5931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76e659a26c6d4f44b015475e20d0befd. Latest consensus state: current_term: 1 leader_uuid: "76e659a26c6d4f44b015475e20d0befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.229182 5931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:59.232630 5681 master_runner.cc:186] Error getting master registration for 127.15.102.253:45841: OK, instance_id { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" instance_seqno: 1781981398542080 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:49:59.252810 5681 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 18:49:59.258313 5702 tablet.cc:2369] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:49:59.262039 5936 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:49:59.262166 5936 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:49:59.262333 5937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:49:59.264354 5937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 18:49:59.273819 5872 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [term 1 FOLLOWER]: Refusing update from remote peer 76e659a26c6d4f44b015475e20d0befd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:59.274521 5822 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } attempt: 5
I20260620 18:49:59.275031 5931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [LEADER]: Connected to new peer: Peer: permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 18:49:59.279675 5851 tablet.cc:2369] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:49:59.291810 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.252037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.15.102.253:45841)
0620 18:49:59.252283 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.252290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.270917 (+ 18627us) server_negotiation.cc:407] Connection header received
0620 18:49:59.271470 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.271479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.271619 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.271703 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.275412 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.276619 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.278921 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.279765 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.279874 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.283995 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.284005 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.284013 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.284064 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.288155 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.289077 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.289084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.289091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.289246 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.289750 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.289756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.289763 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.290171 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.290437 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.291079 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.291378 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 18:49:59 dist-test-slave-zfp8 krb5kdc[5668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981398, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
I20260620 18:49:59.315658 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.252240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.15.102.252:39489)
0620 18:49:59.252438 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.252444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.287270 (+ 34826us) server_negotiation.cc:407] Connection header received
0620 18:49:59.287483 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.287488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.287595 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.287702 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.294616 (+ 6914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.295838 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.299045 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.299819 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.299939 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.306427 (+ 6488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.306436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.306442 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.306481 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.310213 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.311929 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.311937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.311942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.312091 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.312934 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.312940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.312945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.313299 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.313536 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.315032 (+ 1496us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.315286 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 18:49:59.315991 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.275147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:39489 (local address 127.0.0.1:34004)
0620 18:49:59.275547 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 18:49:59.275566 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:49:59.275657 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:49:59.294839 (+ 19182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:49:59.294845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:49:59.295333 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:49:59.295514 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.295527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.299069 (+ 3542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.299075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.300465 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:49:59.300480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.301526 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.301533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:49:59.301779 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.302404 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:49:59.302430 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:49:59.305398 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:49:59.313241 (+ 7843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.313247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.313251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.313735 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.314209 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:49:59.314214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:49:59.314217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:49:59.314417 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:49:59.315014 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:49:59.315020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:49:59.315189 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 18:49:59.315348 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 18:49:59.315624 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261}
I20260620 18:49:59.317358 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.275875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.15.102.252:39489)
0620 18:49:59.293727 (+ 17852us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.293733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.293865 (+ 132us) server_negotiation.cc:407] Connection header received
0620 18:49:59.294140 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.294146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.294653 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.295657 (+ 1004us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.295944 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.297210 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.300659 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.301339 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.305562 (+ 4223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.305847 (+ 285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.305854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.305861 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.305907 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.313081 (+ 7174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.313886 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.313891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.313895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.314059 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.314567 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.314572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.314575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.314885 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.316581 (+ 1696us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.316708 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.316908 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17150,"thread_start_us":16517,"threads_started":1}
I20260620 18:49:59.323300 5822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f: Bootstrap starting.
I20260620 18:49:59.327551 5822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f: Neither blocks nor log segments found. Creating new log.
I20260620 18:49:59.328856 5822 log.cc:824] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f: Log is configured to *not* fsync() on all Append() calls
I20260620 18:49:59.332546 5822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f: No bootstrap required, opened a new log
I20260620 18:49:59.333976 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76e659a26c6d4f44b015475e20d0befd. Latest consensus state: current_term: 1 leader_uuid: "76e659a26c6d4f44b015475e20d0befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.334277 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:49:59.337368 5941 mvcc.cc:205] Tried to move back new op lower bound from 7298995811411902464 to 7298995811220029440. Current Snapshot: MvccSnapshot[applied={T|T < 7298995811411902464}]
I20260620 18:49:59.341876 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.269645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.15.102.254:39339)
0620 18:49:59.269892 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.269900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.289603 (+ 19703us) server_negotiation.cc:407] Connection header received
0620 18:49:59.289791 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.289797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.289879 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.289983 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.307395 (+ 17412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.308639 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.311131 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.312081 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.312216 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.333703 (+ 21487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:49:59.333714 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:49:59.333721 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:49:59.333773 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:49:59.337581 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.338542 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.338548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.338585 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.338863 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:49:59.339394 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:49:59.339399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:49:59.339403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:49:59.339780 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:49:59.340038 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.341263 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.341507 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:49:59.344684 5822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } }
I20260620 18:49:59.345727 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76e659a26c6d4f44b015475e20d0befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.345976 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:49:59.348119 5822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6f12b01d114e878c15d04ccd98426f, State: Initialized, Role: FOLLOWER
I20260620 18:49:59.348819 5822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } }
I20260620 18:49:59.355059 5931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76e659a26c6d4f44b015475e20d0befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.355914 5931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:59.357813 5797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "76e659a26c6d4f44b015475e20d0befd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0b6f12b01d114e878c15d04ccd98426f"
I20260620 18:49:59.358211 5796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:49:59.361125 5937 catalog_manager.cc:1347] Generated new cluster ID: 69ac3dc1a55d4b08945a2827aa15593c
I20260620 18:49:59.361217 5937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:49:59.362138 5796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [term 1 FOLLOWER]: Refusing update from remote peer 76e659a26c6d4f44b015475e20d0befd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:49:59.364951 5797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76e659a26c6d4f44b015475e20d0befd in term 1.
I20260620 18:49:59.357813 5822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:49:59.366768 5931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:49:59.357990 5949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.367908 5949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:49:59.391660 5937 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:49:59.394384 5937 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:49:59.418627 5949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76e659a26c6d4f44b015475e20d0befd. Latest consensus state: current_term: 1 leader_uuid: "76e659a26c6d4f44b015475e20d0befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.418941 5949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:49:59.419045 5919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "76e659a26c6d4f44b015475e20d0befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.419395 5919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd [sys.catalog]: This master's current role is: LEADER
I20260620 18:49:59.420229 5957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "76e659a26c6d4f44b015475e20d0befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76e659a26c6d4f44b015475e20d0befd" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 39339 } } peers { permanent_uuid: "0b6f12b01d114e878c15d04ccd98426f" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 45841 } } peers { permanent_uuid: "486e78b8caa6443abe1b1b9137ff655e" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 39489 } } }
I20260620 18:49:59.420712 5957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:49:59.431541 5937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76e659a26c6d4f44b015475e20d0befd: Generated new TSK 0
I20260620 18:49:59.433463 5937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:49:59.477892 5965 catalog_manager.cc:1259] Loaded cluster ID: 69ac3dc1a55d4b08945a2827aa15593c
I20260620 18:49:59.478103 5965 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f: loading cluster ID for follower catalog manager: success
I20260620 18:49:59.481844 5965 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f: acquiring CA information for follower catalog manager: success
I20260620 18:49:59.485137 5965 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b6f12b01d114e878c15d04ccd98426f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:49:59.598044 5902 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.589752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.15.102.254:39339)
0620 18:49:59.589925 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.589931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.596212 (+ 6281us) server_negotiation.cc:407] Connection header received
0620 18:49:59.596536 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.596542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.596623 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.596695 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.597778 (+ 1083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38684: BlockingRecv error: recv got EOF from 127.0.0.1:38684 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:49:59.599817 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.584230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.15.102.252:39489)
0620 18:49:59.584423 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.584429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.595856 (+ 11427us) server_negotiation.cc:407] Connection header received
0620 18:49:59.598381 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.598387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.598497 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.598565 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.599557 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34020: BlockingRecv error: recv got EOF from 127.0.0.1:34020 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 18:49:59.600848 5903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.582183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.15.102.253:45841)
0620 18:49:59.582379 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.582385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.587821 (+ 5436us) server_negotiation.cc:407] Connection header received
0620 18:49:59.599358 (+ 11537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.599365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.599448 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.599522 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.600579 (+ 1057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53320: BlockingRecv error: recv got EOF from 127.0.0.1:53320 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 18:49:59.648252 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.632354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.15.102.253:45841)
0620 18:49:59.632546 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.632553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.640287 (+ 7734us) server_negotiation.cc:407] Connection header received
0620 18:49:59.640826 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.640833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.641077 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.641212 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:59.641940 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.643124 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.645222 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.645910 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.646002 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.646724 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:59.647424 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:59.647546 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.647776 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.647869 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 18:49:59.695924 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.682520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.15.102.252:39489)
0620 18:49:59.682732 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.682739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.693740 (+ 11001us) server_negotiation.cc:407] Connection header received
0620 18:49:59.694131 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.694138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.694298 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.694396 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.695635 (+ 1239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34024: BlockingRecv error: recv got EOF from 127.0.0.1:34024 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 18:49:59.745920 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.731585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.15.102.253:45841)
0620 18:49:59.731772 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.731778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.739295 (+ 7517us) server_negotiation.cc:407] Connection header received
0620 18:49:59.739806 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.739813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.739905 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.739975 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:59.740660 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.741808 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.743761 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.744444 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.744556 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.745120 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:59.745248 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:59.745302 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.745504 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.745597 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:49:59.793216 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.780036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.15.102.252:39489)
0620 18:49:59.780244 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.780250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.791074 (+ 10824us) server_negotiation.cc:407] Connection header received
0620 18:49:59.791644 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.791651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.791753 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.791833 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.792953 (+ 1120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34026: BlockingRecv error: recv got EOF from 127.0.0.1:34026 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:49:59.812870 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.805347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.15.102.253:45841)
0620 18:49:59.805603 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.805612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.806220 (+ 608us) server_negotiation.cc:407] Connection header received
0620 18:49:59.806605 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.806612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.806697 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.806780 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:59.807577 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.808724 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.810731 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.811429 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.811578 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.812019 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:59.812176 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:59.812276 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.812470 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.812553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:49:59.821964 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.819486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.15.102.252:39489)
0620 18:49:59.819688 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.819694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.820210 (+ 516us) server_negotiation.cc:407] Connection header received
0620 18:49:59.820498 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.820503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.820620 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.820702 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.821708 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34030: BlockingRecv error: recv got EOF from 127.0.0.1:34030 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 18:49:59.841668 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.832761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.15.102.253:45841)
0620 18:49:59.832954 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.832960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.835224 (+ 2264us) server_negotiation.cc:407] Connection header received
0620 18:49:59.835439 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.835446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.835541 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.835664 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:59.836207 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.837527 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.839485 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.840201 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.840348 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.840862 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:59.841015 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:59.841118 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.841293 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.841375 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 18:49:59.853384 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.849039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.15.102.252:39489)
0620 18:49:59.849271 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.849278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.851841 (+ 2563us) server_negotiation.cc:407] Connection header received
0620 18:49:59.852038 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.852045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.852151 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.852264 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.853146 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34038: BlockingRecv error: recv got EOF from 127.0.0.1:34038 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 18:49:59.871883 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.864624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.15.102.253:45841)
0620 18:49:59.864805 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.864811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.865361 (+ 550us) server_negotiation.cc:407] Connection header received
0620 18:49:59.865570 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.865577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.865703 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.865821 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:59.866380 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.867655 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.869629 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.870339 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.870464 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.870939 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:59.871081 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:59.871175 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.871412 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.871503 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 18:49:59.886775 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.884471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.15.102.252:39489)
0620 18:49:59.884703 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.884710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.885227 (+ 517us) server_negotiation.cc:407] Connection header received
0620 18:49:59.885409 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.885415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.885509 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.885633 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.886508 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34052: BlockingRecv error: recv got EOF from 127.0.0.1:34052 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:49:59.913793 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.906391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.15.102.253:45841)
0620 18:49:59.906619 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.906627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.907167 (+ 540us) server_negotiation.cc:407] Connection header received
0620 18:49:59.907496 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.907508 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.907604 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.907713 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:59.908228 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.909438 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.911494 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.912286 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.912422 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.912922 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:59.913072 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:59.913165 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.913379 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.913465 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 18:49:59.923897 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.921641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.15.102.252:39489)
0620 18:49:59.921828 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.921835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.922388 (+ 553us) server_negotiation.cc:407] Connection header received
0620 18:49:59.922545 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.922551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.922665 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.922799 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.923659 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34054: BlockingRecv error: recv got EOF from 127.0.0.1:34054 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:49:59.971225 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:49:59.965126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.15.102.253:45841)
0620 18:49:59.965307 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.965313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.965851 (+ 538us) server_negotiation.cc:407] Connection header received
0620 18:49:59.966035 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.966039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.966133 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.966210 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:49:59.966651 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.967574 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.969414 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:49:59.969857 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:49:59.969966 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:49:59.970448 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:49:59.970557 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:49:59.970624 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:49:59.970799 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 18:49:59.970868 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 18:49:59.979475 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:49:59.977459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.15.102.252:39489)
0620 18:49:59.977684 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 18:49:59.977689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:49:59.978021 (+ 332us) server_negotiation.cc:407] Connection header received
0620 18:49:59.978214 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:49:59.978218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:49:59.978312 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:49:59.978391 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:49:59.979177 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34056: BlockingRecv error: recv got EOF from 127.0.0.1:34056 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 18:50:00.052462 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:00.046851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.15.102.253:45841)
0620 18:50:00.047084 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:00.047091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:00.047430 (+ 339us) server_negotiation.cc:407] Connection header received
0620 18:50:00.047586 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:00.047592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:00.047680 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:00.047781 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:00.048179 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.049315 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.050594 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.051266 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.051393 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:00.051746 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:00.051881 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:00.051962 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:00.052084 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 18:50:00.052167 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:50:00.060299 5943 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:00.058296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.15.102.252:39489)
0620 18:50:00.058492 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:00.058501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:00.058861 (+ 360us) server_negotiation.cc:407] Connection header received
0620 18:50:00.059085 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:00.059097 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:00.059168 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:00.059312 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:00.060051 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34068: BlockingRecv error: recv got EOF from 127.0.0.1:34068 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:50:00.145992 5916 catalog_manager.cc:1259] Loaded cluster ID: 69ac3dc1a55d4b08945a2827aa15593c
I20260620 18:50:00.146093 5916 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: loading cluster ID for follower catalog manager: success
I20260620 18:50:00.148612 5916 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: acquiring CA information for follower catalog manager: success
I20260620 18:50:00.150338 5916 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 486e78b8caa6443abe1b1b9137ff655e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:50:00.198165 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:00.192895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.15.102.253:45841)
0620 18:50:00.193164 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:00.193172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:00.193474 (+ 302us) server_negotiation.cc:407] Connection header received
0620 18:50:00.193743 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:00.193748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:00.193816 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:00.193921 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:00.194365 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.195091 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.196450 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.196885 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.197031 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:00.197445 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:00.197549 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:00.197651 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:00.197819 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 18:50:00.197899 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 18:50:00.208998 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:00.203644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.15.102.252:39489)
0620 18:50:00.203888 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:00.203894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:00.204111 (+ 217us) server_negotiation.cc:407] Connection header received
0620 18:50:00.204242 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:00.204246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:00.204398 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:00.204497 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:00.204823 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.205517 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.206867 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.207350 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.207448 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:00.207900 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:00.208410 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:00.208529 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:00.208598 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 18:50:00.208665 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 18:50:00.222968 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:00.214961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.15.102.252:39489)
0620 18:50:00.215157 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:00.215163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:00.216254 (+ 1091us) server_negotiation.cc:407] Connection header received
0620 18:50:00.216441 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:00.216449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:00.216576 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:00.216698 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:00.217338 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.218740 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.220912 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.221526 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.221650 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:00.222206 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:00.222339 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:00.222432 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:00.222602 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 18:50:00.222680 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 18:50:00.225286 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:00.214909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.15.102.253:45841)
0620 18:50:00.215124 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:00.215133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:00.219366 (+ 4233us) server_negotiation.cc:407] Connection header received
0620 18:50:00.219561 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:00.219567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:00.219663 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:00.219754 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:00.220366 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.221168 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.223278 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.223858 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.223966 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:00.224560 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:00.224682 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:00.224778 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:00.224918 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 18:50:00.224994 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 18:50:00.226315 6071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:00.214812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.15.102.254:39339)
0620 18:50:00.219014 (+ 4202us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:00.219021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:00.219040 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:50:00.219156 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:00.219161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:00.219388 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:00.220301 (+ 913us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:00.220455 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.221386 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.223999 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:00.224596 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:00.224683 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:00.225245 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:00.225666 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:00.225795 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:00.225913 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:50:00.225977 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4073,"thread_start_us":3957,"threads_started":1}
I20260620 18:50:00.231702 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5681
I20260620 18:50:00.232609 5743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:50:00.342343 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5681
W20260620 18:50:00.362829 5847 connection.cc:535] server connection from 127.0.0.1:33986 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 18:50:00.362938 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5755
I20260620 18:50:00.364037 5817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:50:00.480940 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5755
I20260620 18:50:00.501365 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 5830
I20260620 18:50:00.502461 5892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:50:00.611490 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 5830
2026-06-20T18:50:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4396 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 18:50:00 dist-test-slave-zfp8 krb5kdc[6079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 18:50:00 dist-test-slave-zfp8 krb5kdc[6079](info): set up 2 sockets
Jun 20 18:50:00 dist-test-slave-zfp8 krb5kdc[6079](info): commencing operation
krb5kdc: starting...
W20260620 18:50:01.715862 15771 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/test-user.keytab.
Jun 20 18:50:01 dist-test-slave-zfp8 krb5kdc[6079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T18:50:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T18:50:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.102.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
Entry for principal kudu/127.15.102.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab.
I20260620 18:50:01.972738 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:43983
--webserver_interface=127.15.102.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:43983,127.15.102.253:44477,127.15.102.252:36103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 18:50:02.328131 6092 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:50:02.328504 6092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:50:02.328627 6092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:50:02.328758 6092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:50:02.336251 6092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:50:02.336356 6092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:50:02.336447 6092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:50:02.336537 6092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:50:02.336656 6092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:50:02.346653 6092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:43983,127.15.102.253:44477,127.15.102.252:36103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.254.keytab
--principal=kudu/127.15.102.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.254:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.102.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:50:02.348079 6092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:50:02.349414 6092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:50:02.361470 6097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:50:02.361442 6098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:50:02.361510 6100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:50:02.362221 6092 server_base.cc:1034] running on GCE node
I20260620 18:50:02.362952 6092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:50:02.364821 6092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:50:02.366071 6092 hybrid_clock.cc:648] HybridClock initialized: now 1781981402365954 us; error 88 us; skew 500 ppm
I20260620 18:50:02.366432 6092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:50:02 dist-test-slave-zfp8 krb5kdc[6079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981402, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:50:02.369585 6092 init.cc:377] Logged in from keytab as kudu/127.15.102.254@KRBTEST.COM (short username kudu)
I20260620 18:50:02.371140 6092 webserver.cc:460] Webserver started at http://127.15.102.254:45423/ using document root <none> and password file <none>
I20260620 18:50:02.371912 6092 fs_manager.cc:359] Metadata directory not provided
I20260620 18:50:02.372047 6092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:50:02.372373 6092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:50:02.375411 6092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/data/instance:
uuid: "cdfa017204594da986b44979641653af"
format_stamp: "Formatted at 2026-06-20 18:50:02 on dist-test-slave-zfp8"
server_key: "deaefc32cb2b01918daebf4f1132d7b4"
server_key_iv: "043c1280e648ca8b1011ba1b6031946a"
server_key_version: "encryptionkey@0"
I20260620 18:50:02.376212 6092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/wal/instance:
uuid: "cdfa017204594da986b44979641653af"
format_stamp: "Formatted at 2026-06-20 18:50:02 on dist-test-slave-zfp8"
server_key: "deaefc32cb2b01918daebf4f1132d7b4"
server_key_iv: "043c1280e648ca8b1011ba1b6031946a"
server_key_version: "encryptionkey@0"
I20260620 18:50:02.381347 6092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 18:50:02.384964 6107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:50:02.386289 6092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 18:50:02.386469 6092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/wal
uuid: "cdfa017204594da986b44979641653af"
format_stamp: "Formatted at 2026-06-20 18:50:02 on dist-test-slave-zfp8"
server_key: "deaefc32cb2b01918daebf4f1132d7b4"
server_key_iv: "043c1280e648ca8b1011ba1b6031946a"
server_key_version: "encryptionkey@0"
I20260620 18:50:02.386689 6092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:50:02.411154 6092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:50:02.415270 6092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:50:02.415575 6092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:50:02.434175 6092 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.254:43983
I20260620 18:50:02.434226 6158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.254:43983 every 8 connection(s)
I20260620 18:50:02.435619 6092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/data/info.pb
I20260620 18:50:02.439380 6159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:50:02.439702 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 6092
I20260620 18:50:02.439888 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-0/wal/instance
I20260620 18:50:02.440346 15771 external_mini_cluster.cc:1428] Setting key f484d618e1012bbba78495653b18fd9e
I20260620 18:50:02.445904 6159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:50:02 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981402, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.102.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.253@KRBTEST.COM" created.
I20260620 18:50:02.471702 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.449236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.15.102.254:43983)
0620 18:50:02.450091 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:02.450107 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:02.452770 (+ 2663us) server_negotiation.cc:407] Connection header received
0620 18:50:02.452887 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:02.452892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:02.453128 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:02.453338 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:02.455025 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.457086 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.459409 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.459978 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.460122 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:02.464360 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:02.464368 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:02.464381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:02.464415 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:02.467462 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:02.468248 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:02.468255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:02.468257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:02.468401 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:02.468801 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:02.468808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:02.468815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:02.469305 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:02.469515 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:02.470160 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 18:50:02.470416 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":409,"threads_started":1}
I20260620 18:50:02.471771 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.449235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:43983 (local address 127.0.0.1:51896)
0620 18:50:02.450481 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.450526 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:02.452483 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:02.453351 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:02.453357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:02.454185 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:02.454780 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:02.454806 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.457263 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.457272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:02.459191 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:02.459209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.460239 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.460249 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:02.460478 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:02.461180 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:02.461214 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:02.464101 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:02.467601 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:02.467611 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:02.467624 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:02.468106 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:02.468525 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:02.468529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:02.468533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:02.468685 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:02.469453 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:02.469460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:02.469694 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 18:50:02.469978 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 18:50:02.470640 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":85,"thread_start_us":781,"threads_started":1}
I20260620 18:50:02.474881 6159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:50:02.476186 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.475760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59500)
0620 18:50:02.475954 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.476071 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 18:50:02.476493 6108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
I20260620 18:50:02.478061 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.477701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59504)
0620 18:50:02.477871 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.477936 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:50:02.478281 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:44477: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab.
Entry for principal kudu/127.15.102.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab.
I20260620 18:50:02.502054 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.253:44477
--webserver_interface=127.15.102.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab
--principal=kudu/127.15.102.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:43983,127.15.102.253:44477,127.15.102.252:36103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:50:02.527594 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } attempt: 1
I20260620 18:50:02.528824 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.528378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59520)
0620 18:50:02.528568 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.528697 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 18:50:02.529920 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.529458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59536)
0620 18:50:02.529676 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.529762 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260620 18:50:02.530134 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:44477: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
I20260620 18:50:02.598400 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } attempt: 2
I20260620 18:50:02.599805 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.599224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59550)
0620 18:50:02.599555 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.599652 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:50:02.600991 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.600552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59558)
0620 18:50:02.600716 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.600800 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 18:50:02.601187 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:44477: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
I20260620 18:50:02.692461 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } attempt: 3
I20260620 18:50:02.693643 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.693161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59572)
0620 18:50:02.693375 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.693475 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 18:50:02.694789 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.694393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59576)
0620 18:50:02.694588 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.694653 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 18:50:02.694967 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:44477: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
W20260620 18:50:02.827191 6166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:50:02.827581 6166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:50:02.827716 6166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:50:02.827852 6166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:50:02.835026 6166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:50:02.835143 6166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:50:02.835264 6166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:50:02.835353 6166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:50:02.835441 6166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:50:02.838207 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } attempt: 4
I20260620 18:50:02.839484 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.838890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59580)
0620 18:50:02.839207 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.839351 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 18:50:02.840503 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.840164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59594)
0620 18:50:02.840315 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.840390 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:50:02.840698 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.253:44477: Network error: Client connection negotiation failed: client connection to 127.15.102.253:44477: connect: Connection refused (error 111)
I20260620 18:50:02.845806 6166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:43983,127.15.102.253:44477,127.15.102.252:36103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.253.keytab
--principal=kudu/127.15.102.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.253:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.102.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:50:02.847111 6166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:50:02.848448 6166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:50:02.860107 6171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:50:02.860176 6172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:50:02.860152 6174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:50:02.861287 6166 server_base.cc:1034] running on GCE node
I20260620 18:50:02.862083 6166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:50:02.863590 6166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:50:02.864880 6166 hybrid_clock.cc:648] HybridClock initialized: now 1781981402864819 us; error 48 us; skew 500 ppm
I20260620 18:50:02.865242 6166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:50:02 dist-test-slave-zfp8 krb5kdc[6079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981402, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:50:02.868417 6166 init.cc:377] Logged in from keytab as kudu/127.15.102.253@KRBTEST.COM (short username kudu)
I20260620 18:50:02.869776 6166 webserver.cc:460] Webserver started at http://127.15.102.253:35901/ using document root <none> and password file <none>
I20260620 18:50:02.870466 6166 fs_manager.cc:359] Metadata directory not provided
I20260620 18:50:02.870601 6166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:50:02.870915 6166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:50:02.873853 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/data/instance:
uuid: "c13cec004309408a838fb17daf4ff029"
format_stamp: "Formatted at 2026-06-20 18:50:02 on dist-test-slave-zfp8"
server_key: "5b21f0d721bd1a97410d8d98b5bedf1c"
server_key_iv: "1834b30e6c9b3f0a7de5554f6311dc85"
server_key_version: "encryptionkey@0"
I20260620 18:50:02.874642 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/wal/instance:
uuid: "c13cec004309408a838fb17daf4ff029"
format_stamp: "Formatted at 2026-06-20 18:50:02 on dist-test-slave-zfp8"
server_key: "5b21f0d721bd1a97410d8d98b5bedf1c"
server_key_iv: "1834b30e6c9b3f0a7de5554f6311dc85"
server_key_version: "encryptionkey@0"
I20260620 18:50:02.879676 6166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 18:50:02.882856 6181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:50:02.884260 6166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:50:02.884441 6166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/wal
uuid: "c13cec004309408a838fb17daf4ff029"
format_stamp: "Formatted at 2026-06-20 18:50:02 on dist-test-slave-zfp8"
server_key: "5b21f0d721bd1a97410d8d98b5bedf1c"
server_key_iv: "1834b30e6c9b3f0a7de5554f6311dc85"
server_key_version: "encryptionkey@0"
I20260620 18:50:02.884626 6166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:50:02.909680 6166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:50:02.913653 6166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:50:02.913868 6166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:50:02.932148 6166 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.253:44477
I20260620 18:50:02.932168 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.253:44477 every 8 connection(s)
I20260620 18:50:02.933473 6166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/data/info.pb
I20260620 18:50:02.936731 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 6166
I20260620 18:50:02.936920 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-1/wal/instance
I20260620 18:50:02.937209 6233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:50:02.937407 15771 external_mini_cluster.cc:1428] Setting key 710bdafd0b9730bd6b27a7b29f94f536
I20260620 18:50:02.943681 6233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:50:02 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981402, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.102.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.102.252@KRBTEST.COM" created.
I20260620 18:50:02.965744 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.946808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.15.102.254:43983)
0620 18:50:02.947042 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:02.947048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:02.948368 (+ 1320us) server_negotiation.cc:407] Connection header received
0620 18:50:02.950222 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:02.950230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:02.950329 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:02.950450 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:02.951981 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.953162 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.954652 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.955489 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.955642 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:02.959373 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:02.959379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:02.959384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:02.959433 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:02.962552 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:02.963397 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:02.963401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:02.963403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:02.963521 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:02.963984 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:02.963992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:02.963998 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:02.964298 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:02.964458 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:02.965180 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 18:50:02.965413 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 18:50:02.966939 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.947315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:43983 (local address 127.0.0.1:51898)
0620 18:50:02.948184 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.948234 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:02.949935 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:02.950659 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:02.950667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:02.951182 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:02.951810 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:02.951828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.953316 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.953320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:02.954496 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:02.954509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.955698 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.955703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:02.955908 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:02.956672 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:02.956696 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:02.959088 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:02.962676 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:02.962682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:02.962694 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:02.963296 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:02.963637 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:02.963640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:02.963643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:02.963863 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:02.964434 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:02.964443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:02.964809 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 18:50:02.965077 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 18:50:02.965583 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":393,"threads_started":1}
I20260620 18:50:02.968739 6233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:50:02 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981402, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:50:02.984232 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.969546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59596)
0620 18:50:02.969716 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.969736 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:02.969835 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:02.971001 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:02.971006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:02.971487 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:02.971617 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:02.971626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.973140 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.973147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:02.974044 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:02.974056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.974919 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.974925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:02.975134 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:02.975820 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:02.975839 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:02.978106 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:02.981444 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:02.981457 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:02.981462 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:02.981836 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:02.982261 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:02.982266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:02.982271 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:02.982405 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:02.983058 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:02.983064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:02.983217 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 18:50:02.983422 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 18:50:02.983731 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260620 18:50:02.984280 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.969831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.15.102.253:44477)
0620 18:50:02.970546 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:02.970553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:02.970582 (+ 29us) server_negotiation.cc:407] Connection header received
0620 18:50:02.970630 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:02.970635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:02.970815 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:02.971001 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:02.971803 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.972994 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.974189 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:02.974764 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:02.974916 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:02.978325 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:02.978336 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:02.978350 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:02.978384 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:02.981294 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:02.981990 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:02.981999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:02.982001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:02.982121 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:02.982528 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:02.982536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:02.982539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:02.982900 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:02.983144 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:02.983587 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 18:50:02.983845 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":527,"thread_start_us":375,"threads_started":1}
I20260620 18:50:02.987082 6233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:50:02.988346 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.987887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37946)
0620 18:50:02.988099 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.988198 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 18:50:02.988665 6183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
I20260620 18:50:02.990525 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:02.990127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37950)
0620 18:50:02.990274 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:02.990349 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 18:50:02.990818 6233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.102.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab.
Entry for principal kudu/127.15.102.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab.
I20260620 18:50:02.994714 15771 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
/tmp/dist-test-taskc6hSqA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.252:36103
--webserver_interface=127.15.102.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab
--principal=kudu/127.15.102.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.102.212:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.102.254:43983,127.15.102.253:44477,127.15.102.252:36103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 18:50:03.040160 6233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 1
I20260620 18:50:03.041576 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.040861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37952)
0620 18:50:03.041144 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.041310 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 18:50:03.042816 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.042299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37954)
0620 18:50:03.042553 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.042665 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 18:50:03.043067 6233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
I20260620 18:50:03.111368 6233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 2
I20260620 18:50:03.112654 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.112054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37962)
0620 18:50:03.112303 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.112430 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 18:50:03.113797 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.113367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37966)
0620 18:50:03.113530 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.113625 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 18:50:03.114089 6233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
I20260620 18:50:03.139978 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } attempt: 5
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981402, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:50:03.153947 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.140851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59612)
0620 18:50:03.141049 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.141073 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:03.141221 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:03.141613 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:03.141620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:03.142029 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:03.142214 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.142223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.143231 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.143239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.144425 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.144447 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.145171 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.145176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.145393 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.146087 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:03.146117 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:03.148227 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:03.151376 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.151381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.151385 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.151750 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.152138 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.152143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.152147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.152367 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.152942 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:03.152947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:03.153104 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 18:50:03.153262 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 18:50:03.153591 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 18:50:03.153947 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.140962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59612 (local address 127.15.102.253:44477)
0620 18:50:03.141180 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.141186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.141208 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:50:03.141396 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.141400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.141465 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.141573 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.142371 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.143081 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.144585 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.145018 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.145155 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.148426 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.148431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.148434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.148466 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.151169 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.151870 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.151878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.151882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.152008 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.152483 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.152489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.152492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.152792 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.152952 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.153376 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.153598 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 18:50:03.155313 6159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:50:03.156451 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.156076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37978)
0620 18:50:03.156246 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.156327 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 18:50:03.157407 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.157071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37980)
0620 18:50:03.157238 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.157303 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:50:03.157665 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
I20260620 18:50:03.205384 6233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 3
I20260620 18:50:03.206691 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.206230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37990)
0620 18:50:03.206443 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.206534 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 18:50:03.207926 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.207501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:37998)
0620 18:50:03.207747 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.207820 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260620 18:50:03.208189 6233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
I20260620 18:50:03.208877 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 1
I20260620 18:50:03.209883 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.209474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38004)
0620 18:50:03.209656 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.209740 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:50:03.210795 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.210469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38010)
0620 18:50:03.210611 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.210692 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 18:50:03.211004 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
I20260620 18:50:03.279347 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 2
I20260620 18:50:03.280532 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.280024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38020)
0620 18:50:03.280287 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.280372 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 18:50:03.281513 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.281171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38028)
0620 18:50:03.281358 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.281417 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 18:50:03.281705 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
W20260620 18:50:03.336450 6240 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 18:50:03.336863 6240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 18:50:03.337011 6240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 18:50:03.337165 6240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 18:50:03.344633 6240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 18:50:03.344769 6240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 18:50:03.344889 6240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 18:50:03.344997 6240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 18:50:03.345117 6240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 18:50:03.351435 6233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 4
I20260620 18:50:03.352654 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.352184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38036)
0620 18:50:03.352413 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.352502 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 18:50:03.353598 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.353283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38052)
0620 18:50:03.353425 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.353497 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 18:50:03.353811 6233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
I20260620 18:50:03.355872 6240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.102.212:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.102.254:43983,127.15.102.253:44477,127.15.102.252:36103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/krb5kdc/kudu_127.15.102.252.keytab
--principal=kudu/127.15.102.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.102.252:36103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.102.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 17:42:20 UTC on ccycloud.quasar-rymxqh.root.comops.site
build id 4457681
ASAN enabled
I20260620 18:50:03.357429 6240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 18:50:03.358878 6240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 18:50:03.371014 6245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:50:03.370946 6246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 18:50:03.370986 6248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 18:50:03.371745 6240 server_base.cc:1034] running on GCE node
I20260620 18:50:03.372607 6240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 18:50:03.374001 6240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 18:50:03.375305 6240 hybrid_clock.cc:648] HybridClock initialized: now 1781981403375191 us; error 42 us; skew 500 ppm
I20260620 18:50:03.375633 6240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981403, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 18:50:03.378819 6240 init.cc:377] Logged in from keytab as kudu/127.15.102.252@KRBTEST.COM (short username kudu)
I20260620 18:50:03.380262 6240 webserver.cc:460] Webserver started at http://127.15.102.252:34273/ using document root <none> and password file <none>
I20260620 18:50:03.380981 6240 fs_manager.cc:359] Metadata directory not provided
I20260620 18:50:03.381116 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 18:50:03.381435 6240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 18:50:03.384464 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/data/instance:
uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b"
format_stamp: "Formatted at 2026-06-20 18:50:03 on dist-test-slave-zfp8"
server_key: "66eb617fc82a00f5808622e3734133b6"
server_key_iv: "a40b435339288fe5861a04fc42f20b83"
server_key_version: "encryptionkey@0"
I20260620 18:50:03.385249 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/wal/instance:
uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b"
format_stamp: "Formatted at 2026-06-20 18:50:03 on dist-test-slave-zfp8"
server_key: "66eb617fc82a00f5808622e3734133b6"
server_key_iv: "a40b435339288fe5861a04fc42f20b83"
server_key_version: "encryptionkey@0"
I20260620 18:50:03.387913 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 3
I20260620 18:50:03.389050 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.388538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38068)
0620 18:50:03.388736 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.388841 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 18:50:03.390344 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.389816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38074)
0620 18:50:03.390090 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.390193 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260620 18:50:03.390604 6159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.102.252:36103: Network error: Client connection negotiation failed: client connection to 127.15.102.252:36103: connect: Connection refused (error 111)
I20260620 18:50:03.391224 6240 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260620 18:50:03.394747 6255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 18:50:03.396092 6240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 18:50:03.396263 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/data,/tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/wal
uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b"
format_stamp: "Formatted at 2026-06-20 18:50:03 on dist-test-slave-zfp8"
server_key: "66eb617fc82a00f5808622e3734133b6"
server_key_iv: "a40b435339288fe5861a04fc42f20b83"
server_key_version: "encryptionkey@0"
I20260620 18:50:03.396477 6240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 18:50:03.418720 6240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 18:50:03.423220 6240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 18:50:03.423560 6240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 18:50:03.442785 6240 rpc_server.cc:307] RPC server started. Bound to: 127.15.102.252:36103
I20260620 18:50:03.442802 6306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.102.252:36103 every 8 connection(s)
I20260620 18:50:03.444139 6240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/data/info.pb
I20260620 18:50:03.447671 6307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 18:50:03.449746 15771 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu as pid 6240
I20260620 18:50:03.449928 15771 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc6hSqA/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781981043007005-15771-0/minicluster-data/master-2/wal/instance
I20260620 18:50:03.450371 15771 external_mini_cluster.cc:1428] Setting key 4cc14b55e2002adfaaac08c9596b199c
I20260620 18:50:03.456369 6307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981403, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:50:03.473788 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.452396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38090 (local address 127.15.102.252:36103)
0620 18:50:03.453220 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.453235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.453290 (+ 55us) server_negotiation.cc:407] Connection header received
0620 18:50:03.455634 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.455663 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.456177 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.456610 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.457952 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.459715 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.461221 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.462063 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.462239 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.466426 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.466433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.466445 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.466473 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.469442 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.470270 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.470276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.470282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.470503 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.470920 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.470924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.470926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.471505 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.471746 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.472302 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.472747 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":300,"threads_started":1}
I20260620 18:50:03.474112 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.459228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51902 (local address 127.15.102.254:43983)
0620 18:50:03.459464 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.459470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.460294 (+ 824us) server_negotiation.cc:407] Connection header received
0620 18:50:03.460468 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.460473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.460553 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.460661 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.461715 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.462644 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.464103 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.464522 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.464651 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.468262 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.468267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.468270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.468302 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.471127 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.471966 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.471970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.471972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.472158 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.472638 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.472641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.472644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.472884 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.473053 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.473529 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.473748 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:50:03.474216 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.459279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.254:43983 (local address 127.0.0.1:51902)
0620 18:50:03.460135 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.460195 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:03.460339 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:03.460710 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:03.460715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:03.461290 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:03.461556 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.461565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.462798 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.462806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.463947 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.463959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.464752 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.464760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.464926 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.465549 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:03.465578 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:03.468068 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:03.471297 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.471308 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.471315 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.471841 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.472305 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.472311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.472314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.472463 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.473049 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:03.473057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:03.473238 (+ 181us) client_negotiation.cc:769] Sending connection context
0620 18:50:03.473412 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 18:50:03.473729 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":417,"threads_started":1}
I20260620 18:50:03.475735 6307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981403, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.254@KRBTEST.COM
I20260620 18:50:03.490971 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.476478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.253:44477 (local address 127.0.0.1:59626)
0620 18:50:03.476663 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.476688 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:03.476799 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:03.477207 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:03.477212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:03.477664 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:03.477800 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.477808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.479103 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.479108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.480314 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.480325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.481258 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.481262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.481469 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.482015 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:03.482056 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:03.484513 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:03.488271 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.488275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.488278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.488758 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.489254 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.489259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.489262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.489417 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.489937 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:03.489942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:03.490149 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 18:50:03.490313 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 18:50:03.490596 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 18:50:03.491034 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.476722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.15.102.253:44477)
0620 18:50:03.476905 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.476910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.476929 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:50:03.476990 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.476995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.477072 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.477172 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.477950 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.478972 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.480452 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.481103 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.481265 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.484706 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.484711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.484715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.484748 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.488144 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.488904 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.488907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.488910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.489110 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.489527 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.489534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.489538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.489817 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.489971 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.490466 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.490646 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 18:50:03.492324 6307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } has no permanent_uuid. Determining permanent_uuid...
I20260620 18:50:03.493021 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.476908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51912 (local address 127.15.102.254:43983)
0620 18:50:03.477042 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.477046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.477151 (+ 105us) server_negotiation.cc:407] Connection header received
0620 18:50:03.477296 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.477302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.477377 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.477480 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.478812 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.479851 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.481360 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.481989 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.482110 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.486881 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.486890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.486897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.486938 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.489986 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.490818 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.490826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.490830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.490989 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.491566 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.491571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.491574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.491854 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.492136 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.492445 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.492668 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981403, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.252@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.102.253@KRBTEST.COM
I20260620 18:50:03.507486 6311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.493055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38102)
0620 18:50:03.493315 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.493334 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:03.493461 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:03.493824 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:03.493827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:03.494243 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:03.494370 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.494378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.495578 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.495583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.496970 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.496982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.497898 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.497905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.498170 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.498733 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:03.498756 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:03.501505 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:03.504684 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.504690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.504694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.505258 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.505673 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.505676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.505679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.505812 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.506440 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:03.506444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:03.506581 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 18:50:03.506719 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 18:50:03.507032 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 18:50:03.507687 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.493104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.15.102.252:36103)
0620 18:50:03.493282 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.493288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.493392 (+ 104us) server_negotiation.cc:407] Connection header received
0620 18:50:03.493604 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.493612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.493693 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.493797 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.494505 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.495450 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.497104 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.497724 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.497863 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.501691 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.501695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.501699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.501727 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.504542 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.505376 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.505381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.505385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.505547 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.505928 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.505935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.505941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.506311 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.506467 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.506851 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.507287 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":74}
I20260620 18:50:03.511307 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.494905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.15.102.253:44477)
0620 18:50:03.495076 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.495079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.495093 (+ 14us) server_negotiation.cc:407] Connection header received
0620 18:50:03.495130 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.495133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.495184 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.495330 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.496825 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.497811 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.499574 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.500156 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.500282 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.505305 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.505310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.505335 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.505377 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.508582 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.509315 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.509320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.509323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.509490 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.509852 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.509858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.509861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.510186 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.510460 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.510705 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.510931 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:50:03.513391 6307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: Bootstrap starting.
I20260620 18:50:03.516688 15771 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 18:50:03.517412 6307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: Neither blocks nor log segments found. Creating new log.
I20260620 18:50:03.518749 6307 log.cc:824] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: Log is configured to *not* fsync() on all Append() calls
I20260620 18:50:03.521967 6307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: No bootstrap required, opened a new log
I20260620 18:50:03.519748 6134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cdfa017204594da986b44979641653af"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51912
I20260620 18:50:03.533318 6307 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } }
I20260620 18:50:03.533733 6307 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1a7fff18bf4ac4b25d7279bb39e23b, State: Initialized, Role: FOLLOWER
I20260620 18:50:03.534235 6307 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } }
I20260620 18:50:03.535121 6313 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.535518 6313 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:50:03.536283 6307 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 18:50:03.545467 6324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 18:50:03.545611 6324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 18:50:03.543640 6240 master_runner.cc:186] Error getting master registration for 127.15.102.254:43983: OK, instance_id { permanent_uuid: "cdfa017204594da986b44979641653af" instance_seqno: 1781981402428842 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:50:03.551563 6240 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:50:03.567867 6159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 4
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981402, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.254@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
I20260620 18:50:03.585955 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.568894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38112 (local address 127.15.102.252:36103)
0620 18:50:03.569116 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.569123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.569146 (+ 23us) server_negotiation.cc:407] Connection header received
0620 18:50:03.569348 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.569355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.569461 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.569571 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.570614 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.571879 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.573773 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.574593 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.574732 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.578883 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.578890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.578896 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.578937 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.582698 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.583612 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.583618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.583622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.583781 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.584220 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.584225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.584229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.584693 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.584863 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.585349 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.585584 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 18:50:03.586000 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.568760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38112)
0620 18:50:03.568993 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.569016 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:03.569168 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:03.569653 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:03.569659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:03.570208 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:03.570449 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.570461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.572059 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.572065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.573613 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.573629 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.574776 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.574785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.575045 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.575837 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:03.575862 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:03.578671 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:03.582854 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.582861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.582866 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.583462 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.583907 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.583913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.583917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.584090 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.584842 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:03.584848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:03.585045 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 18:50:03.585219 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 18:50:03.585537 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 18:50:03.591662 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af: Bootstrap starting.
I20260620 18:50:03.595021 6159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af: Neither blocks nor log segments found. Creating new log.
I20260620 18:50:03.596241 6159 log.cc:824] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af: Log is configured to *not* fsync() on all Append() calls
I20260620 18:50:03.600212 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af: No bootstrap required, opened a new log
I20260620 18:50:03.608072 6159 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } }
I20260620 18:50:03.608522 6159 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfa017204594da986b44979641653af, State: Initialized, Role: FOLLOWER
I20260620 18:50:03.609076 6159 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } }
I20260620 18:50:03.610244 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.610653 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:50:03.611590 6159 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:50:03.611702 6134 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 18:50:03.611939 6134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:50:03.614614 6134 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } }
I20260620 18:50:03.617136 6282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdfa017204594da986b44979641653af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b"
I20260620 18:50:03.617709 6282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:50:03.621479 6282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdfa017204594da986b44979641653af in term 1.
I20260620 18:50:03.622169 6134 leader_election.cc:288] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [CANDIDATE]: Term 1 election: Requested vote from peers c13cec004309408a838fb17daf4ff029 (127.15.102.253:44477), eb1a7fff18bf4ac4b25d7279bb39e23b (127.15.102.252:36103)
I20260620 18:50:03.622291 6109 leader_election.cc:302] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdfa017204594da986b44979641653af, eb1a7fff18bf4ac4b25d7279bb39e23b; no voters:
I20260620 18:50:03.622774 6327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 18:50:03.629848 6327 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [term 1 LEADER]: Becoming Leader. State: Replica: cdfa017204594da986b44979641653af, State: Running, Role: LEADER
I20260620 18:50:03.630637 6327 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } }
I20260620 18:50:03.638300 6332 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdfa017204594da986b44979641653af. Latest consensus state: current_term: 1 leader_uuid: "cdfa017204594da986b44979641653af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.638866 6332 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: This master's current role is: LEADER
I20260620 18:50:03.646733 6337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 18:50:03.653016 6233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } attempt: 5
I20260620 18:50:03.663205 6337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 18:50:03 dist-test-slave-zfp8 krb5kdc[6079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781981402, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.102.253@KRBTEST.COM for kudu/127.15.102.252@KRBTEST.COM
W20260620 18:50:03.668355 6113 tablet.cc:2369] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 18:50:03.675709 6261 tablet.cc:2369] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 18:50:03.680546 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.653865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.102.252:36103 (local address 127.0.0.1:38118)
0620 18:50:03.654414 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 18:50:03.654440 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 18:50:03.654578 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 18:50:03.655217 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 18:50:03.655223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 18:50:03.655796 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 18:50:03.656050 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.656064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.657665 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.657671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.659117 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 18:50:03.659133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.660529 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.660563 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 18:50:03.660973 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.661900 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 18:50:03.661961 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0620 18:50:03.666411 (+ 4450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 18:50:03.672377 (+ 5966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.672409 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.672414 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.673335 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.674192 (+ 857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 18:50:03.674201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 18:50:03.674206 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 18:50:03.674544 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 18:50:03.675880 (+ 1336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 18:50:03.675885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 18:50:03.676269 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 18:50:03.677159 (+ 890us) client_negotiation.cc:240] Negotiation successful
0620 18:50:03.678900 (+ 1741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371}
I20260620 18:50:03.680651 6346 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 18:50:03.680626 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.654436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.15.102.252:36103)
0620 18:50:03.654632 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.654640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.654665 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:50:03.654739 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.654745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.654827 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.654966 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.656309 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.657443 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.659359 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.660112 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.660259 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.666738 (+ 6479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.666746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.666752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.666801 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.672134 (+ 5333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.673537 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.673544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.673573 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.673873 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.674714 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.674748 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.674752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.675623 (+ 871us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.676592 (+ 969us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.677371 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.678993 (+ 1622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 18:50:03.689177 6233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029: Bootstrap starting.
I20260620 18:50:03.694648 6233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029: Neither blocks nor log segments found. Creating new log.
I20260620 18:50:03.684765 6092 master_runner.cc:186] Error getting master registration for 127.15.102.253:44477: OK, instance_id { permanent_uuid: "c13cec004309408a838fb17daf4ff029" instance_seqno: 1781981402926779 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 18:50:03.695305 6092 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 18:50:03.696192 6233 log.cc:824] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029: Log is configured to *not* fsync() on all Append() calls
I20260620 18:50:03.701810 6282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [term 1 FOLLOWER]: Refusing update from remote peer cdfa017204594da986b44979641653af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:50:03.703065 6332 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:50:03.712536 6233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029: No bootstrap required, opened a new log
I20260620 18:50:03.735878 6233 raft_consensus.cc:348] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } }
I20260620 18:50:03.736346 6233 raft_consensus.cc:729] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13cec004309408a838fb17daf4ff029, State: Initialized, Role: FOLLOWER
I20260620 18:50:03.736898 6233 consensus_queue.cc:260] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } }
I20260620 18:50:03.737006 6313 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdfa017204594da986b44979641653af. Latest consensus state: current_term: 1 leader_uuid: "cdfa017204594da986b44979641653af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.737329 6313 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:50:03.739418 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.698116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.15.102.253:44477)
0620 18:50:03.698320 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.698327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.705774 (+ 7447us) server_negotiation.cc:407] Connection header received
0620 18:50:03.706704 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.706710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.706799 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.706872 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.720391 (+ 13519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.721607 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.723686 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.724310 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.724429 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.728791 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.728798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.728803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.728846 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.732578 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.734060 (+ 1482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.734068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.734075 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.734330 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.735049 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.735057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.735060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.735652 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.736033 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.738449 (+ 2416us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.738830 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:50:03.742063 6345 mvcc.cc:205] Tried to move back new op lower bound from 7298995829542735872 to 7298995829278617600. Current Snapshot: MvccSnapshot[applied={T|T < 7298995829542735872}]
I20260620 18:50:03.750017 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdfa017204594da986b44979641653af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.750360 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: This master's current role is: LEADER
I20260620 18:50:03.750783 6352 mvcc.cc:205] Tried to move back new op lower bound from 7298995829542735872 to 7298995829278617600. Current Snapshot: MvccSnapshot[applied={T|T < 7298995829542735872}]
I20260620 18:50:03.751734 6337 catalog_manager.cc:1347] Generated new cluster ID: a83de6538d334985bb4468b89bc81b03
I20260620 18:50:03.751837 6337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 18:50:03.764683 6354 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.766767 6233 sys_catalog.cc:564] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 18:50:03.768721 6207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [term 0 FOLLOWER]: Advancing to term 1
I20260620 18:50:03.769501 6354 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:50:03.769794 6313 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdfa017204594da986b44979641653af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.770058 6313 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:50:03.772450 6207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [term 1 FOLLOWER]: Refusing update from remote peer cdfa017204594da986b44979641653af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 18:50:03.773404 6332 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 18:50:03.778770 6208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdfa017204594da986b44979641653af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c13cec004309408a838fb17daf4ff029"
I20260620 18:50:03.779515 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.698485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.15.102.254:43983)
0620 18:50:03.698642 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.698648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.719766 (+ 21118us) server_negotiation.cc:407] Connection header received
0620 18:50:03.721675 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.721683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.721772 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.721850 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.731224 (+ 9374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.732351 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.749315 (+ 16964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.750085 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.750188 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.772052 (+ 21864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.772059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.772064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.772113 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.775927 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.776941 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.776947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.776950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.777120 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.777595 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.777601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.777604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.777998 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.778264 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.778852 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.779123 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 18:50:03.788657 6337 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 18:50:03.790560 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:03.706720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.15.102.252:36103)
0620 18:50:03.706918 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.706926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.719930 (+ 13004us) server_negotiation.cc:407] Connection header received
0620 18:50:03.720185 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.720191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.720326 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.720444 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.728422 (+ 7978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.729598 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.738877 (+ 9279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:03.739749 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:03.739886 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:03.764260 (+ 24374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 18:50:03.764267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 18:50:03.764271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 18:50:03.764316 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 18:50:03.768096 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.784334 (+ 16238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.784341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.784344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.784552 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 18:50:03.785386 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 18:50:03.785396 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 18:50:03.785399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 18:50:03.785788 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 18:50:03.786066 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:03.789831 (+ 3765us) server_negotiation.cc:299] Negotiation successful
0620 18:50:03.790112 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 18:50:03.791088 6337 catalog_manager.cc:1504] Loading token signing keys...
I20260620 18:50:03.807865 6337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af: Generated new TSK 0
I20260620 18:50:03.809011 6337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 18:50:03.813555 6364 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdfa017204594da986b44979641653af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.814060 6364 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:50:03.814416 6354 sys_catalog.cc:455] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdfa017204594da986b44979641653af. Latest consensus state: current_term: 1 leader_uuid: "cdfa017204594da986b44979641653af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.814669 6354 sys_catalog.cc:458] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 18:50:03.815138 6208 raft_consensus.cc:2396] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cdfa017204594da986b44979641653af for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 18:50:03.817404 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdfa017204594da986b44979641653af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfa017204594da986b44979641653af" member_type: VOTER last_known_addr { host: "127.15.102.254" port: 43983 } } peers { permanent_uuid: "c13cec004309408a838fb17daf4ff029" member_type: VOTER last_known_addr { host: "127.15.102.253" port: 44477 } } peers { permanent_uuid: "eb1a7fff18bf4ac4b25d7279bb39e23b" member_type: VOTER last_known_addr { host: "127.15.102.252" port: 36103 } } }
I20260620 18:50:03.817816 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdfa017204594da986b44979641653af [sys.catalog]: This master's current role is: LEADER
I20260620 18:50:03.878207 6374 catalog_manager.cc:1259] Loaded cluster ID: a83de6538d334985bb4468b89bc81b03
I20260620 18:50:03.878335 6374 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029: loading cluster ID for follower catalog manager: success
I20260620 18:50:03.882102 6374 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029: acquiring CA information for follower catalog manager: success
I20260620 18:50:03.884543 6374 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c13cec004309408a838fb17daf4ff029: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 18:50:03.992355 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:03.977396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.15.102.252:36103)
0620 18:50:03.977572 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.977579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.990417 (+ 12838us) server_negotiation.cc:407] Connection header received
0620 18:50:03.990629 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.990635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.990725 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.990824 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.992076 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38130: BlockingRecv error: recv got EOF from 127.0.0.1:38130 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 18:50:03.994612 6237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:03.977464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.15.102.253:44477)
0620 18:50:03.977685 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.977691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.992990 (+ 15299us) server_negotiation.cc:407] Connection header received
0620 18:50:03.993277 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.993284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.993384 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.993455 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.994378 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59646: BlockingRecv error: recv got EOF from 127.0.0.1:59646 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 18:50:03.996984 6163 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:03.981147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.15.102.254:43983)
0620 18:50:03.981340 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:03.981346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:03.995373 (+ 14027us) server_negotiation.cc:407] Connection header received
0620 18:50:03.995609 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:03.995615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:03.995716 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:03.995796 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:03.996746 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51936: BlockingRecv error: recv got EOF from 127.0.0.1:51936 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 18:50:04.060611 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.039374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.15.102.253:44477)
0620 18:50:04.039573 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.039580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.051844 (+ 12264us) server_negotiation.cc:407] Connection header received
0620 18:50:04.052233 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.052240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.052478 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.052626 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.053432 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.054636 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.057227 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.057922 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.058019 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.058846 (+ 827us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.059656 (+ 810us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.059833 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.060116 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.060227 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 18:50:04.125936 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:04.113399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.15.102.252:36103)
0620 18:50:04.113611 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.113618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.123884 (+ 10266us) server_negotiation.cc:407] Connection header received
0620 18:50:04.124107 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.124114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.124287 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.124422 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:04.125653 (+ 1231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38132: BlockingRecv error: recv got EOF from 127.0.0.1:38132 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 18:50:04.195416 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.175681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.15.102.253:44477)
0620 18:50:04.175907 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.175916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.187888 (+ 11972us) server_negotiation.cc:407] Connection header received
0620 18:50:04.188160 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.188166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.188272 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.188389 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.189144 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.190519 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.192749 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.193482 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.193612 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.194225 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.194395 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.194479 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.194971 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.195052 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 18:50:04.225183 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:04.222768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.15.102.252:36103)
0620 18:50:04.223013 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.223021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.223604 (+ 583us) server_negotiation.cc:407] Connection header received
0620 18:50:04.223840 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.223845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.223937 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.224040 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:04.224950 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38148: BlockingRecv error: recv got EOF from 127.0.0.1:38148 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:50:04.237771 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.230414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.15.102.253:44477)
0620 18:50:04.230595 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.230601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.231213 (+ 612us) server_negotiation.cc:407] Connection header received
0620 18:50:04.231452 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.231459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.231555 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.231673 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.232184 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.233382 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.235545 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.236183 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.236313 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.236874 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.237027 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.237127 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.237298 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.237391 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 18:50:04.247252 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:04.245245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.15.102.252:36103)
0620 18:50:04.245437 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.245443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.245465 (+ 22us) server_negotiation.cc:407] Connection header received
0620 18:50:04.245648 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.245654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.245753 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.245838 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:04.246942 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38160: BlockingRecv error: recv got EOF from 127.0.0.1:38160 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 18:50:04.262959 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.255306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.15.102.253:44477)
0620 18:50:04.255520 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.255526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.256067 (+ 541us) server_negotiation.cc:407] Connection header received
0620 18:50:04.256267 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.256273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.256377 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.256487 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.256972 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.258347 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.260363 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.261233 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.261397 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.261889 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.262061 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.262180 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.262342 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.262435 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260620 18:50:04.277542 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:04.269976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.15.102.252:36103)
0620 18:50:04.270270 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.270277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.275899 (+ 5622us) server_negotiation.cc:407] Connection header received
0620 18:50:04.276090 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.276096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.276213 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.276346 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:04.277280 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38170: BlockingRecv error: recv got EOF from 127.0.0.1:38170 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
I20260620 18:50:04.296749 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.289000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.15.102.253:44477)
0620 18:50:04.289209 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.289215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.289758 (+ 543us) server_negotiation.cc:407] Connection header received
0620 18:50:04.290045 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.290052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.290160 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.290289 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.290866 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.292114 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.294137 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.295001 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.295149 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.295695 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.295910 (+ 215us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.296042 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.296277 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.296381 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 18:50:04.307024 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:04.304442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.15.102.252:36103)
0620 18:50:04.304626 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.304633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.305199 (+ 566us) server_negotiation.cc:407] Connection header received
0620 18:50:04.305398 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.305404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.305510 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.305660 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:04.306770 (+ 1110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38182: BlockingRecv error: recv got EOF from 127.0.0.1:38182 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 18:50:04.336751 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.329085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.15.102.253:44477)
0620 18:50:04.329291 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.329297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.329885 (+ 588us) server_negotiation.cc:407] Connection header received
0620 18:50:04.330132 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.330144 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.330239 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.330353 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.330874 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.332166 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.334368 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.335046 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.335194 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.335810 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.336013 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.336124 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.336306 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.336410 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 18:50:04.347164 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:04.345078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.15.102.252:36103)
0620 18:50:04.345273 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.345279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.345304 (+ 25us) server_negotiation.cc:407] Connection header received
0620 18:50:04.345654 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.345661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.345763 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.345879 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:04.346880 (+ 1001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38194: BlockingRecv error: recv got EOF from 127.0.0.1:38194 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 18:50:04.392733 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.385155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.15.102.253:44477)
0620 18:50:04.385353 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.385360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.385927 (+ 567us) server_negotiation.cc:407] Connection header received
0620 18:50:04.386178 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.386184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.386276 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.386376 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.386888 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.388166 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.390444 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.391255 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.391382 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.391868 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.392017 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.392116 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.392297 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.392406 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 18:50:04.402043 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:04.400143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.15.102.252:36103)
0620 18:50:04.400312 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.400318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.400337 (+ 19us) server_negotiation.cc:407] Connection header received
0620 18:50:04.400506 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.400513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.400611 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.400687 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:04.401796 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38204: BlockingRecv error: recv got EOF from 127.0.0.1:38204 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 18:50:04.475482 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.470535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.15.102.253:44477)
0620 18:50:04.470764 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.470770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.470957 (+ 187us) server_negotiation.cc:407] Connection header received
0620 18:50:04.471133 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.471137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.471223 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.471356 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.471760 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.472507 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.473829 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.474258 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.474357 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.474726 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.474831 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.474898 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.475090 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.475152 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 18:50:04.483320 6310 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 18:50:04.481562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.15.102.252:36103)
0620 18:50:04.481810 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.481832 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.482045 (+ 213us) server_negotiation.cc:407] Connection header received
0620 18:50:04.482165 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.482168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.482242 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.482324 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 18:50:04.483053 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38218: BlockingRecv error: recv got EOF from 127.0.0.1:38218 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 18:50:04.547813 6324 catalog_manager.cc:1259] Loaded cluster ID: a83de6538d334985bb4468b89bc81b03
I20260620 18:50:04.547936 6324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: loading cluster ID for follower catalog manager: success
I20260620 18:50:04.550452 6324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: acquiring CA information for follower catalog manager: success
I20260620 18:50:04.552376 6324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb1a7fff18bf4ac4b25d7279bb39e23b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 18:50:04.619956 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.615184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.15.102.253:44477)
0620 18:50:04.615442 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.615448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.615718 (+ 270us) server_negotiation.cc:407] Connection header received
0620 18:50:04.615873 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.615877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.615945 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.616026 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.616411 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.617099 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.618390 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.618788 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.618875 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.619222 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.619354 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.619419 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.619570 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.619632 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 18:50:04.631089 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.625678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.15.102.252:36103)
0620 18:50:04.625839 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.625844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.626191 (+ 347us) server_negotiation.cc:407] Connection header received
0620 18:50:04.626331 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.626335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.626482 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.626597 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.626911 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.627672 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.629177 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.629561 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.629656 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.630084 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.630575 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.630689 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.630785 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.630845 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 18:50:04.648015 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.637290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.15.102.253:44477)
0620 18:50:04.637530 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.637538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.641872 (+ 4334us) server_negotiation.cc:407] Connection header received
0620 18:50:04.642046 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.642054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.642193 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.642300 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.643251 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.644291 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.646138 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.646580 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.646714 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.647217 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.647385 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.647486 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.647656 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.647747 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 18:50:04.649044 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.637315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.15.102.252:36103)
0620 18:50:04.637561 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.637569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.642691 (+ 5122us) server_negotiation.cc:407] Connection header received
0620 18:50:04.642932 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.642938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.643043 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.643127 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.643756 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.644524 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.646892 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.647563 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.647675 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.648317 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.648437 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.648562 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.648680 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.648756 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 18:50:04.649703 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 18:50:04.636698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.15.102.254:43983)
0620 18:50:04.641510 (+ 4812us) server_negotiation.cc:206] Beginning negotiation
0620 18:50:04.641516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 18:50:04.641537 (+ 21us) server_negotiation.cc:407] Connection header received
0620 18:50:04.641674 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 18:50:04.641679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 18:50:04.641903 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 18:50:04.642073 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 18:50:04.643112 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.644291 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.646875 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 18:50:04.647439 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 18:50:04.647555 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 18:50:04.648465 (+ 910us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 18:50:04.648931 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 18:50:04.649045 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 18:50:04.649348 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 18:50:04.649430 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4663,"thread_start_us":4569,"threads_started":1}
I20260620 18:50:04.654906 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 6092
I20260620 18:50:04.655913 6154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:50:04.766036 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 6092
I20260620 18:50:04.787158 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 6166
I20260620 18:50:04.788345 6228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:50:04.895051 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 6166
I20260620 18:50:04.915017 15771 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu pid 6240
I20260620 18:50:04.916155 6302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 18:50:05.022905 15771 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc6hSqA/build/asan/bin/kudu with pid 6240
2026-06-20T18:50:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4410 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8807 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361981 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST